It sounds to me like everyone did the right thing here. You respected their wishes, his wife respected his feelings, and he was calm in how he handled asking you to stop. If you had already "crossed the Delaware" and started pounding her I don't understand why he chose then to halt things, but...