I wouldn't jump to conclusions about the neighbors until you meet them first. Introduce yourself, be cordial, and offer to provide assistance if any deer they shoot make it onto your property. Heck, even go so far as to contact them whenever you find a carcass on your property and ask them if they lost a buck recently and offer to give the rack back to them. If you do that, they more than likely will reciprocate they courtesy to you.
Sure, there are some that will still refuse to cooperate, and if so, continue to do as I've suggested above. But if they make it clear that you should not contact them to retrieve a deer shot on the property you have permission to hunt, then honor their request. It's your job to ensure you make a clean, effective kill on an animal WHICH YOU CAN RECOVER. Shoot one right on the property line as it's headed for a fence crossing to the neighbors is a shot you should never have taken in the first place if you don't have permission to cross over the fence to retrieve.
And also, you probably don't know the history of the property you just received permission to hunt. There may have been numerous incidents with poachers/ tresspassers in past years, and the neighbors are doing the landowner a favor by contacting the authorities when unusual activity is going on there.
