Pressing the E button just doesn't feel as intuitive to me for some reason, especially where the cursor is used to highlight various items. Especially things like doors.
I agree with you, but some people like E, or maybe something else. That's why Hotkey should be high priority !
With my experience with Terraria, I usually found left action clicking to be a bit inprecise , and have very often used items by accident when trying to access a menu.
Mouse pointing + E doesn't make sence for me too. It should be: - pointing + right click - E alone for the stuff behind you At least, when you are close to a door and pointing nowhere, E should open the door (usefull when you run firing your bow).