Party Problems

Three of my friends and I decided to play through Starbound together. One of us made a server is it is working great, hosting four people with no lag. There is only one problem though, the party system doesn't work. Whenever any of us invite each other to the party, the Accept/Decline window pops up, but when we accept we aren't added. As of now, three of us have met up at the outpost, but we can't do the Erichius Crystal mission without partying up. Has this happened to anyone and doesn't anyone know the fix? Thanks for any help!