Before you entered the lifestyle with your wife, did you have lots of communication and lay down ground rules including stopping? If not, then you better do it right away. Without ground rules your marriage could go anywhere
And if it is not working out the way you want or envisioned, then...