It sounds ideal, except it might not be all it seams, if he is fucking her and your wondering if to confront her whats to say he has not told her you blow him? so maybe she might confront you. As mentioned above does she know you are Bi?
I suppose a lot depends on if you trust him, but i know from experience that best mates can also be double crossers too. What would your wifes reaction be if he told her about you?
Have you thought that maybe this could become an arrangement more open to all three of you, then you could play together, then there is the problem where would his wife fit into this?.
Its a nice problem to have if you can keep it going securely or include you wife into this.