I think the key is really when you find someone who respects the relationship and understands the desires of the couple. It would also be the problem with the wife who, if she is not truly faithful, starts by leaving you out of the game.
Anyway, it is a difficult road, leaving fantasies...