Thanks for the replies guys. I like the idea of moving into "his" spot after he left and trying to call him back. That's pretty much basic turkey hunting 101 and it never crossed my mind. I'm going to say it's because it was my first hunt of the year and I'm not in the swing of things yet.
Getting on his level would've been awesome, but without crossing property lines it would've been difficult. Maybe not impossible though. As far as getting really aggressive and risking bumping him, I had 2 more days to hunt, there's not a ton of birds around, and I was on a small parcel. We have a couple large parcels in the area but most of the land we can hunt is 80 acres here, 45 acres there, etc.