How will this work exactly?
I think I have a pretty good idea of what AI can do for software engineering, because I use it for that nearly every day and I experiment with different models and IDEs.
The way that has worked for me is to make prompts very specific, to the point where the prompt itself would not be comprehensible to someone who's not in the field.
If you sat a rando with no CS background in front of Cursor, Windsurf or Claude code, what do you suppose would happen?
It seems really doubtful to me that overcoming that gap is "just more training", because it would require a qualitatively different sort of product.
And even if we came to a point where no technical knowledge of how software actually works was required, you would still need to be precise about the business logic in natural language. Now you're writing computer code in natural language that will read like legalese. At that point you've just invented a new programming language.
Now maybe you're thinking, I'll just prompt it with all my email, all my docs, everything I have for context and just ask it to please make my boss happy.
But the level of integrative intelligence, combined with specialized world knowledge required for that task is really very far away from what current models can do.
The most powerful way that I've found to conceptualize what LLMs do is that they execute routines from huge learnt banks of programs that re-combine stored textual information along common patterns.
They're cut and paste engines where the recombination rules are potentially quite complex programs learnt from data.
This view fits well with the strengths and weaknesses of LLMs - they are good at combining two well understood solutions into something new, even if vaguely described.
But they are quite bad at abstracting textual information into a more fundamental model of program and world state and reasoning at that level.
I strongly suspect this is intrinsic to their training, because doing this is simply not required to complete the vast majority of text that could realistically have ended up in training databases.
Executing a sophisticated cut&paste scheme is in some ways just too effective; the technical challenge is how do you pose a training problem to force a model to learn beyond that.
I just completed a prototype of a non-trivial product that was vibe-coded just to test the ability and limits of LLMs.
My experience aligns largely with your excellent comment.
>But the level of integrative intelligence, combined with specialized world >knowledge required for that task is really very far away from what current >models can do.
Where LLMs excel are to put out large templates of what is needed, but they are frayed at the edges. Imagine programming as a jigsaw puzzle where the pieces have to fit together. LLMs can align the broader pieces, but fail to fit them precisely.
>But they are quite bad at abstracting textual information into a more >fundamental model of program and world state and reasoning at that level.
The more fundamental model of program is a "theory" or "mental-model" which unfortunately is not codified in the training data. LLMs can put together broad outlines based on their training data, but lack the precision in modeling at a more abstract level. For example, how concurrency could impact memory access is not precisely understood by the LLM - since it lacks a theory of it.
> the technical challenge is how do you pose a training problem to force a model > to learn beyond that.
This is the main challenge - how can an LLM learn more abstract patterns. For example, in the towers of hanoi problem, can the LLM learn the recursion and what recursion means. This requires LLM to learn abstraction precisely. I suspect LLMs learn abstraction "fuzzily" but what is required is to learn abstraction "precisely". The precision or determinism is largely where there is still a huge gap.
LLM-boosters would point to the bitter lesson and say it is a matter of time before this happens, but I am a skeptic. I think the process of symbolism or abstraction is not yet understood enough to be formalized.