Without more information, hard to provide insight... are you married or in a relationship?
It's not all doom and gloom, as some may suggest. It is completely possible for your hot wife partner and her husband to enjoy, and flourish in a polyamorous situation - this is more common than one might think.
The potential downside isn't all on you. If the husband and wife can't handle it, most of it's on them. I have had several situations where a wife has developed emotional attachment, asking for "more," where "more" ranged from requesting a live-in arrangement to her divorcing her husband for me. The two most complicated situations have resulted in long-term affairs; in one case, I've fathered two children with my affair partner, in the other, she's never been able to form lasting relationships with a man other than me. The former woman has now been married and divorced four times, the latter having recently experienced the failure of her second engagement.
Neither woman can get past the emotional attachment they have for me. My situation is further complicated by a long-term marriage, one that I don't plan to end, for a slew of reasons. So we (me and my affair partners) are at sort of a stalemate.