Agreed. But also, comparison and competition between individuals is only one of the ways in which improvement happens. Consider for example that it's also possible to build something for personal consumption and iteratively improve on the design without regard for what anyone else thinks of it. Cooking comes to mind.
Right. But even that is shaped directly or indirectly by environment you live in. The way you scratch your own itch looks differently depending on what itch you have. Plus, humans are social animals, we live in groups and constantly judge each other and try to have others judge us favorably.
AI has none of that now - it only gets direct human feedback from those controlling the training (or at a second level, the harness), and that feedback is really in service of the humans at the steering wheels. Sum total of humanity, mixed in the blender, and flavored to make the trainers look good in front of their peers.
Now, if AI could interact directly and propagate that feedback to their training, or otherwise learn on-line, that changes. It's a qualitative jump. The second one is, once there's enough AIs interacting with human economy and society directly, that their influence starts to outweigh ours. At that point, they'll end up evolving their own standards and benchmarks, and then it's us who will be judged by their measure.
(I.e. if you think we have it bad now, with how we're starting to adapt our writing and coding style to make it easier for LLMs, just wait when next-gen models start participating in the economy, and we'll all be forced by the market forces to learn some weird, emergent token-efficient English/Chinese pidgin that AI-run companies prefer their suppliers to use.)