My macbook air seems to be unable to run the unstable nightly branch of starbound. I am currently using the fully updated Mountain Lion OS X. The issue specifically is that when I start the game the icon pops up but disappears in less that a second and the game says it is running for 1/2 a second. Point is that the game never actually opens or runs.
I'm seeing this also on a Mac mini. I'm running Mac OS X v10.9, the game launches and then immediately quits. You might check here http://playstarbound.com/help/ and also see if disabling the steam overlay helps. I'm going to try that tonight and see if that fixes my problem.