Bug/Issue Can't connect to host in current multiplayer - 'Connection Failed'. Connects fine to a new game.

We've tried all kinds of combinations of restarts to game, steam, router and computer and different ways of inviting (code, lan, steam) we still get the message. However, when we tested on a new game we were able to connect to each other just fine. Neither of us are getting any errors in SMAPI and when we check the logs it shows 'lobby entered' and I'm there for 43 seconds and then it says 'disconnecting from server'. We were playing on lan fine yesterday, not sure what could be causing it since we see no errors and it's fine on a new game.