Congrats on a good all around deer rifle. Your expectations are where they need to be. NEF Handi's are not bench rifle nor bolt gun equivilants, but they are a great 200 yard and under deer rifle. They are very finicky when it comes to ammo. You will have to try a few to see what it likes. My 243 liked the Federal Fusion very well. Also if you hand load,, the NEF barrels are not to friendly to hot loads. Staying on the bottom end of the load range is suggested. Also a trick to get it to group a little better if not already is to put a rubber spacer on the barrel forarm post to help float the barrel better.
Other than that you should be able to get at least under 2" group this rifle with the right load.