That is a hard one to answer, as it depends on what you need to do. If all you need to do is mow, lightly disk / till, and plant already existing plots you could easily get by with something in the 25 HP range (of course, it would take longer because your attachments would be sized to match the tractor). If you are talking about clearing, plowing or chisel plowing, and spreading tons of lime / fertilizer or doing quite a few acres of plots, 40 HP or more would be more reasonable.
However, you have heard right about the 4WD tractors being able to do more work than a comparable HP older 2WD ag tractor. I can easily plow with a 3-bottom 14" plow with my 50 HP 4WD Kubota in old fields that hadn't been plowed in almost 50 years. In fact, I have buried my plow just about to the top link a couple of times when I wasn't paying attention (unfortunately, the majority of new 4WD "compact" tractors don't have draft control on the 3 point hitch). The tractor didn't stall out - it just stopped and all 4 wheels were spinning but i have R4 tires on it, not ag tires.
Hope this helps!