> Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features.

Anyone with sufficiently good taste in how to program effectively and architect will disagree with you on this. The short leash method is how you ensure good results when you're functioning outside of the training data. If you're even a modestly above average programmer this is afaik the only way to ensure fast, quality development with LLMs.

> This again feels outdated. I think we're mving towards humans no longer needing to understand a codebase, and letting AI drive it.

I think you are perhaps unaware of a world of programming where AI is still woefully inept. I have observed very consistently in all languages with manual memory management frequent issues with handling it. Trust me, it's not as simple as sticking it in a loop with Valgrind.