The skeleton stops growing somewhere around 2 or 3 years, but the rack size may peak out at 4, 5 8, or 9 depending on the deer. I think 4.5 is the "textbook" definition of a mature buck. That's when they become insanely elusive and really show how smart an older deer can be.
Side tracked here, but I've seen/heard it said way too often that an old doe is the smartest deer in the woods. Having killed far more old does than old bucks, and having observed behavior of hundreds of does, I have to say that they are nowhere near as smart as an old buck. Yes they are very keen, and good at picking out a hunter in the woods. But the difference is that the mature buck (4.5+) will never show his face, while the mature doe will stomp around snd make a scene.