HELP! Xbox Co-op account corrupt!

Started a Co-op account with 2 other people on Xbox 1 played about 40 hours worth logging on and off with each other no problem then today I log on (I am the first farmhand that joined) AND THERE IS IN ROOM ON THE SERVER! Now the second farmhand joined and he HAS ACCESS TO MY CHARACTER and I don't! He somehow has the file of my character and I do not. Please help! I have no idea how this happened and I do not want to delete my character and lose all of the progress on that farm.