We've done this exact same thing (just once) with a couple we used to swing with.
It was a bit anticlimactic for me as the most exciting part of our lifestyle is watching my wife with other men (as opposed to me having sex with another woman). It was fun being with his wife alone, but I missed...