It sounds like you really want this guy to fuck your wife but just haven't admitted it. From the way you describe her and your thoughts, I'd want to fuck her too. If I was your neighbor, I'd already be fucking your wife. You have several options.
1. Do nothing and let nature take it's course...