Years ago, I was trying a combination that my Traditions 54 cal would shoot well. I had the exact same problem you are reporting when I was shooting patch and ball. It would have a 5 inch pattern at 50 yards for the first 2 shots, then it would tighten to a 1 inch pattern for the next 2. After a complete cleaning, it was back to the poor pattern.
I gave up on the patch and ball and went to a conical. I shot probably 5 or 6 different conicals until I came across the Hornady Great Plains bullet. Huge bullet, too! Over 500gr.! It patterns about 1/2 inch groups at 75 yards, now.
The point is that you have to find what the gun likes. It can be pricey, too, if you don't find it fast.