you will not affect the genetics of the herd if you remove them. you will not affect the genetics of the herd if you let them live.
You NEED some mature bucks on the property to suppress the younger bucks from doing most of the breeding. A 1.5 y/o buck actively participating in the rut will lose enough weight his 2nd fall/ winter that he will NEVER express his genetic antler potential. That has been proven over and over in penned deer studies.
The flip side to the coin is that you can have TOO MANY mature bucks as well. Usually, the older bucks are more dominant, fight harder with other bucks, and are MUCH more likely to break off one of your mature 'trophy' bucks. No biggie that year, as the trophy will have an even bigger rack the following year, but if the pedicle is damaged, that trophy buck will never be a trophy again in subsequent years. Even worse than damaging the antlers, there is a very real possibility of your 'trophy' mature buck getting seriously injured from fighting.
So how do you find that balance between a few mature bucks left in the herd to help improve expressed genetics down the road in younger deer versus too many mature bucks left in the herd causing injuries/ damage directly to other mature bucks??? Most in TN will never have that problem unless they are behind a high fence or only remove an extremely small percentage of the available bucks (and their neighbors are doing the same). I'm not sure if Ames is in that boat or not. I know that my farms ARE at that point. I'm still amazed at how much success I've had with my 1 buck limit (actually 1 'buck of choice' plus 1 cull if available) I instituted about 5 or 6 years ago. This year, I've seen a 5.5 y/o cull I chose to pass, a 4.5 y/o I chose to kill, and 2 other 4.5 y/o's I was forced to pass (because I had already killed my 'buck of choice'). I am certainly seeing the young bucks not participate in the rut (aside from the silly chasing at the end of October), but I'm also seeing more and more bucks each year broken or with injured pedicles.