Ball seals better when seated, it is hardened and the seat is not, so the seat gets shaped to fit. You get better oil pressure at idle (no leakage). The plunger does not seat well unless it's lapped in, and they seldom are.
There is also the story spread by suppliers and others that the 9/16" ball was used in the Cooper S. I have owned a few S motors over the years and never found one. it may have been a homologated part for competition, I dunno.

I currently use a ball, it works fine when fitted as I described above.

ps. These days I prefer the plunger too. The only thing the ball has over it is it won't get jammed with metal swarf, but if you have swarf circulating you have bigger problems.

Might be worth trying a different gauge also.