So why doesn't the mouse pointer work that way on an Apple trackpad?
Surely if that's the case then when you move your finger to the upper left then the pointer should move to the bottom right. Because that's how it would work if it was a real object and you were pushing the pointer around with your finger. Why is scrolling a special case?
Honestly though, I wouldn't mind that much if Apple hadn't decided to call it "natural" scrolling, like you're weird if you prefer up for scroll up and down for scroll down. It's both smug and reeks of the same kinda of discriminatory attitude that made life hard for left handers.
The pointer represents your (pointer) finger. Single-finger motions affect the pointer. Scrolling motions affect a representation of the document. When I move my (real) finger up, my (real) finger moves up. When I put my (real) finger on a (real) document and move it up, the paper of the document moves up, causing new text from the bottom to appear in my field of view.
Or scrolling represents a viewpoint, such as a frame, or a finger pointing at the document. Then it's the other way round again. You can't just declare what the metaphor is, it's arbitrary.