Dump your lover - the statement that you don't want other men is because you love your lover. Dump him, deepen your relationship with your husband and move on, once over it, to a purely sexual relationship with another man.
Or - dump your husband and go to your lover. But say goodbye to cukold...