You've missed the point.

The point doesn’t seem to have been thought through.

The point is, if road engineers changed their process and materials, and to you it felt like driving on the same road, with the same wear and tear and potholes, you wouldn't even notice.

If AIs can generate code that looks ridiculous to humans but over time has the correct performance, the correct behaviour, no-one outside of software engineers will know or care.

> The point is, if road engineers changed their process and materials,

They do those in labs, and then studies are made to prove that it can replace the current composition. They do not invent those on the spot and let the drivers QA the road.

> If AIs can generate code that looks ridiculous to humans but over time has the correct performance, the correct behaviour

It’s on you to prove that this big “if” can be realized. A -> B only matters when A is true.