Compute is finite but human stupidity is limitless. You can ask AI to do something unimaginably stupid and it will spend its full capacity trying to satisfy your stupid request, which may not even be possible.
For those of us with practical budgets not measured in the millions, that bar for stupidity is pretty damn low, and I’m not convinced that there’s enough compute being built to account for every idiot in the world.
What does human stupidity have to do with this? If you're too stupid to ask it a good prompt then you're probably too stupid to design something for it to do.
I'm saying ask the AI something beyond your intellectual capabilities. Or ask it something beyond your intrinsic knowledge base. It can produce a working machine that does the job, but you won't be able to understand it.
You can still verify the machine works by using the machine... but you won't necessarily understand it. Similar to how you own a smartphone... you know it works, but you will never fully understand everything about it.