I have problems joining certain people also. For instance I enjoy playing with ReaverofSouls77 but cannot join his party. So I have to join someone that is playing in his group.
I experienced this with the zombie maps also so I believe it may be partially due to a person's router settings.