Bug/Issue Serious game freezing/Lag Spikes whenever another player comes close

I can't quite seem to find anything on this. Maybe I'm using the wrong words when I'm looking, maybe I'm being an idgit. In essence, my issue is that I will be hit with a bad lag spike/freeze/whatever it is whenever I come close to my friend, who is hosting the session through Steam Multiplayer. Oftentimes, this freeze will last for a few seconds -rarely causing a crash- and then update to have everything that happened in those few seconds having already occurred. In cases where we are going through potentially hazardous areas, this usually ends in my severe maiming or demise. According to said friend, he does not have these issues unless I am hosting for him, and will see my freezing as me simply sitting still and not reacting at all to whatever is happening in the world.

I've had this issue between computers both high and low power, and since I have had the game if memory serves. And it irritates me royally that I can't figure out why this happens or how to fix/alleviate it.