Karate Lord Farmer? (Keep breaking stone walls)

Apparently my farmer is a karate master. I've set up my farm with stone walls and enclosed exits so that if I forget to close one door, I have another one keeping the critters in. One of them is facing east/west and my farmer has a tendency to chop down the stone walls with any item but the watering can and food selected. Even if the cursor box is on an empty tile he will still break the wall. The sad part is most of the time I just lose the wall piece all together. And I'm right clicking to open the gate. Just wanted to make it aware that this is a possible bug. On another note, he can't break the gate piece, so that's a thing