Xbox -Coop Disconnect

Everytime I load up a world or a New one on co-op and when I invite people in one of my world. The World exits and kicks me out has an error saying cannot connect to world. But When I join their world it Works Just Fine. I TRYED Resetting Game, New World, Reset Internet, Called Internet Provider and even tryed Hardwired It's still not working and it is getting annoying, I wanna be able to play coop on my own farms so I can play when my friend Not on. I don't understand why it's happening, especially since I can join other peoples world's with no issue.