This is a dated take to me. I think the next stage in unlocking productivity is so called "loop engineering"; figuring out how to effectively not read all the code while ensuring quality. To me that means implementing statistical quality control and formal methods.
Before I get there I have to figure out how to reliably audit plan adherence. The problem is that when the specs are in natural language, as they are, you need a fallible LLM to verify it.
We built a conversational terraform platform at work, you describe what you need for your infra and it produces the terraform code following company standards and rules. What is important for us is the LLM never writes any code. It just extracts a structured spec from the conversation, and a deterministic engine (plain python, no LLM) renders the code from approved modules only. So there is no "code quality" discussion at all, and since the spec is structured and not natural language, you don't need another LLM to verify it, the risk moves to the extraction step which is small and easy to eval. But probably this only works in narrow domains where the output is composable from approved blocks (IaC, config etc), and its not a new idea, just old discipline behind a chat interface. But in that slice the speedup is way beyond 2x.
tf is yaml anyway, and llms can output it natively in structured output mode.
Sure, structured output gives you valid syntax, but thats not the problem we are solving. Our engine simply cannot produce anything outside of the approved module registry (pinned versions, mandatory tags etc). LLM with structured output still can produce plausible but wrong configs. Also same spec always renders the same code, thats what reviewers and auditors actually care
I agree that removing human from the loop is the next stage, but we're not there yet beyond small programs.
Agents left alone tend to create so much tech debt, that once the program becomes so messy that they can't fix one bug without creating two new bugs, it's too late to even clean that up.
The program will be super tidy in superficial aspects that linters catch (everything neatly formatted and verbosely commented), and roughly appear to do what it's supposed to do, but everything in between will be "I can't even".
We need something else than formal methods, because the problem is usually in lack of simplicity - you get four versions of the problem solved in four times in four in different ways, each uniquely flawed and just incompatible enough with the others that unifying them is too big and hairy for the agent, and will result in eight different glue adapters written in the process.
This will be a deeply unpopular opinion, but I have an 80k line Go project I've created where I haven't read the source, but would absolutely stake its quality against most hand-written projects of a similar size[0].
Recent LLMs struggle with the author's stated issues only inline: they're entirely capable of going back and evaluating codebases to find architectural issues and LLM slop signatures, especially when you use other models to check one model's output. This wasn't true until Fable-class models, but it's true now.
0: https://github.com/pjlsergeant/byre