Really fun idea! Though I agree about the feedback on the physics: the ball doesn't travel as far as I'd like and bounces off the edge too often.

The game also often wanted to reorient me towards the flag which was annoying when I was trying to go another direction.

It also sometimes was tricky to drag the ball - It thought I was trying to pan the perspective.

All that said, this is fucking rad! I think with a few small tweaks you've got something really special. And it could easily be extended with windmills, power-ups, etc. I'm excited to play more and see it progress!