I've noticed the same exact thing. From 3.5-4.5 is the biggest jump I see in antler growth.
Actually, there is a much larger percentage gain
from 2 1/2 to 3 1/2, and from 3 1/2 to 4 1/2 just hugely adds even more!
What most commonly happens in most TN areas being hunted is
that the very top-end 2 1/2's rarely if ever survive to 3 1/2.
I'm talking typically about 2 1/2's that gross score 115 to 135.
These specific "top-end" young bucks have almost zero chance to survive for a variety of reasons.
#1: They are young and inexperienced to human dangers.
#2: Their antlers often appear 140 class when they're actually 115 class.
This is because their skeletal system has finished growing, i.e. small deer makes antlers "look" larger.
#3: They get specifically targeted and hunted for, while below averaged antlered young bucks never get a 2nd thought.
#4: Their antlers actually appear to meet the "trophy" status pre-defined by many hunters,
such as "120-class, 140-class", etc.
#5: There are also many hunters who set their bar at "8 points, or 10 points, etc.", which essentially means most only above average 2 1/2's & 3 1/2's are specifically targeted at "trophy" bucks.
#6: They typically rut roam @ 2-plus miles in every direction around whatever might be their most "core" area. This means that even with a 1,000-acre area of control, one of these bucks you refer to as "yours", may actually spend more of his rut time outside the boundaries of your property.
#7: These way above average youngsters appear to "feel their oats" more, are often "bullies", and seem to be much more on the move during daylight than below average antlered ones. This means they not only get killed more by selective hunters, but also are more likely to run over by a car, or killed in a fight with another buck.
The sad reality may be that a top-end antlered young buck has better odds of surviving to 4 1/2 on most TN public lands than he has on most "intensely" managed private lands, where antler restrictions are used to determine which bucks to "protect". Most antler restrictions actually
INCREASE the rate of hunter harvests of these very best antlered young bucks.