So it does make a difference to you who initiated the idea. Now that you've had some experience, if it had been you that brought up the idea would you have the same attitude towards him being with anyone else?
How is your husband involved (if he is) in your version of the lifestyle?