Hah yeah, I felt like that when I first started on this fantasy. I think over time the desire to see it happen has increased while thinking about it and masturbating to the fantasy has desensitized me to the negative feelings / jealous associated with it.
I think part of this arose from a time when we broke up prior to getting engaged, where I felt sick but also oddly aroused thinking she might having sex with other men. When we got back together and she initially down-played that anything happened, then over the years admitted she had fucked a guy, then later two. I'd always kind of known it and finding out made me feel sort of bad but also highly aroused and set me onto the wife-sharing fantasy. For a long time, it was a mix of jealousy and arousal but now its mostly just arousal. Nothing has happened yet but she seems increasingly open to the idea.
Good luck in your journey.