It is so annoying. I'm standing on the edge of the ocean or whatever and facing at and cast my rod and when i do my character changes direction and starts casting onto the land.
I don't know about mobile or controller, but on PC... If you click a tile by your feet your farmer will turn that direction.