When one has history with a particular buck, it's sometimes possible to know that buck's exact age. Having experienced this a few times over the years in my hunting environments, I can tell you that a buck of a particular age will often "appear" (live, in the field) either a year younger or a year older than his actual age, and this appearance is often effected by the angle of your view, the date, and just how much viewing you get.
Among bucks I've killed that were not previously known, I've made more mistakes in pulling the trigger on bucks that ended up being younger rather than older, than what I had field guessed their ages. In each of these circumstances, I felt I'd have been less disappointed had I chosen to let that buck walk, rather than pull the trigger on him. Not to say I wasn't happy, as I consider any deer taken home a true gift to be appreciated. But it's also a great "gift" to get to watch one grow to full maturity, and then take him home.
Very healthy 2 1/2-yr-old bucks will often "look" like a 3 1/2 during the rut. Very healthy 3 1/2's can often "look" like a 4 1/2 during the rut. However, each buck is an individual, and I've seen 4 1/2-yr-old bucks, during the rut, that had the body appearance more in line with an average 2 1/2. The generalities hold, but there will always be exceptions, whether we're aging by looking at teeth or body profiles.
After having killed several top-end 2 1/2's which I believed to be 3 1/2's, I finally decided to simply not pull the trigger on any buck I thought was younger than 4 1/2. That resulted in mistakes being at least 3 1/2. Over the past few years, I've gone back & forth between trying to take bucks no younger than either 4 1/2 or 5 1/2. The last buck I killed was aged at 5 1/2 or older, and I believe he was most likely 6 1/2.
One of the problems in waiting to 5 1/2 is that the 4 1/2-yr-old you pass this year, might shift his range next year, and you might never have another opportunity at him before he dies, even if that's by natural causes. And of course, there is always the possibility he could simply die of whatever cause around 4 1/2.
During the 2014 season, I had the opportunity to take a 4 1/2-yr-old 140-class mainframe 8-pointer that I had been tracking since he was 2 1/2. His routine had changed little from 2 1/2 to 3 1/2 and then to 4 1/2. Having focused more of my attention on a larger antlered 5 1/2 or older buck (which I did kill during that 2014 muzzleloader season), I decided that "nice" 140-class 4 1/2 would be allowed to walk (at least by me, although I had absolutely zero problem with anyone else wanting to take him).
Going into the 2015 season, that "nice" mainframe 8, which would now be 5 1/2, became one of my most targeted bucks.
Surprisingly, not only did I fail to get a trail cam pic which could positively identify him (even during the summer, early fall), but I never saw him. I suspect he most likely died of some natural cause, although it's possible he was killed miles away and I never knew. However, I've had similar to happen before, and a buck I thought had died, suddenly reappears a year or two after he was presumed deceased.
I believe most hunter/mangers are not realizing just how far and wide most bucks range, and can periodically shift their ranges, often way beyond the hunter's property lines or hunting area. The best thing is they often do come back to a previous pattern. So a particular buck you assumed someone else killed after you passed him up as a 3 1/2, he just might surprise you by showing back up as a 5 1/2 right about where you had passed him up as a 3 1/2.
Enjoy the journey, take what makes you happy, and don't worry about what you can't control.
You control your trigger finger, and sometimes little more.