I have few comments on what you wrote , nothing to judge , just what I understood from what you wrote..
your hubby have the fantasy of his wife cheating, however, that was not expressed to you except when he was ......…
you didn’t discuss that with him, and it seems things went where you now...