I once had the same thoughts while watching my exwife during a group sex session at a friend's organised swing party. He and his wife Bev had been swinging for years and were teaching us the ropes. My wife was in the middle of a group of 3 males, sucking, fucking and being groped, and Bev was in a separate room having fun with some other guys. I asked Dave what made us do what we do and he told me "it's like owning a Ferrari, she is beautiful, sexy and everyone wants to drive it and to let others drive it gives great pleasure to both the borrower and the owner because that Ferrari is yours and not theirs".
It clarified things for me so I hope it helps you.
Also he did say that it's fucking horny aswell lol