I have noticed lately that sometimes when you try to join an invite or join someone that is in a pregame lobby, you actually lag the other people out you are trying to join. There have been nights when just as the game was about to start I got connection interruption and boom I'm out of the lobby. I actually did this to scooby about a week ago when trying to join one of his games. I got into the lobby which was filled with PCP clan members and no scooby. One of them piped up hey we just lost someone and another said it was scooby.
The same thing happened to me last night. I invited Dani to a game and then we joined a lobby. Just as the game was about to start I lagged out. A few seconds later Dani jumped into the main menu with me. She said she wasn't able to join the game because it already started. I asked if she tried to join me and she said yes.
So be aware that if you are lagging out of games, it might be because your friends are trying to join your session.