If you're on XB1, hold the guide button down.
If missed the opportunity to do that and you know who invited you and they are in your friends list, you can go to your friends list and join from there.
Yeah, I'm not sure if there's a way to see standing invites anywhere. I know the 360 would have them as messages and you could join from there.