I'd leave them be unless you're just infested with them. They won't outgrow oaks. They're generally an understory tree and when they die standing they often become hollow, which makes excellent homes for all sorts of birds, flying squirrels, etc. By all means if they actually are out competing oaks then thin them out but otherwise they are an excellent food source for all sorts of animals, including deer. In my woods they get choked out and die by the time they get around 15" dbh, but the ground is littered with saplings that deer, turkeys, bunnies, squirrels, and grouse back when I still had them all seem to like it.