Can't open doors or anything 90% of the time

EDIT: I found the problem! My mouse wasn't on top of the door I was trying to open. I didn't realize that was the magic key because so many times before I kept the mouse far away and was still able to access the doors. But oh well, at least I know now.

I don't know if it's just my game or if everyone experiences this, but I often cant talk to anyone, open chests, doors, go up ladders, etc. until I've spent a few minutes trying to do the action. I will walk up to a door, hit the button to open it, and nothing happens. Then, after 10 tries, I finally get in.

This was just annoying and a waste of time before (because I literally will spend an in game hour trying to get into my house) but now its really affecting gameplay. Sometimes, I'll be in the mine, notice its 12 AM, and then rush to the ladder. But it normally doesnt allow me to use it for 30-60 minutes. So then finally, after ramming my character into the same spot over and over, I get to the surface, dash home, and then spend the same amount of time trying to open the door. Then, at 1:40 I'm climbing into bed... and nothing happens. I jump out, run back in, over and over until it's 2 AM and i've passed out, losing energy and money.

Does anyone else have this issue? The first time it happened, I laughed it off. Second time wasn't as funny. After the 10th time, its definitely getting old. I've tried changing controls, leaving my hands empty, filling them up, standing a block away (which actually does work sometimes) pressing right up against the door, etc. and then after i've done each of these things a couple times, it finally works. What can I do to fix this?

I am playing on a mac