I agree , it might incrementally optimize itself very well, but i think for now at least anything super innovative will still come from a human that can think beyond a few steps. There are surely far better possible architectures, training methods etc that would initially lead to worse performance if approached stepwise.
Yeah, anyone who's seen it trying to improve code could tell you what that optimization looks like.
Oh, this part is taking too long, let's replace it with an empty function.
Oh wait, now it's not working, let's add the function.
Oh, this part is taking too long...
It would be hilarious if this world wasn't full of idiots.