> Is Coding → Prompting like Assembly → High Level Coding? […] I do not agree with this simile. Compilers are, for the most part, deterministic in a way that current AI tools are not.
It’s not quite about the determinism. It’s about being able to reason about the relationship between source code and compiled program with formal precision. You can predict which changes in the source code will lead to which exact changes in the behavior of the compiled program. The same isn’t the case about changes to an LLM prompt and the LLM’s output.
You could make an AI deterministic by fixing its source of randomness. That still wouldn’t allow you to reason about how its output will change when (for example) you add or remove a word in the prompt. The only way to find out is to run the LLM (= have the prompt run through the model and observe what comes out).
That is the fundamental difference. Changes to source code have predictable and reason-able outcomes. You generally don’t have to compile the code and test it to know how precisely the change will affect the behavior of the compiled program according to the semantics of the programming language. That’s the case even if the compiler uses some probabilistic heuristics for trade-offs in code generation, and hence isn’t deterministic on the machine code level.
To repeat, the difference is how you can reason about a compiler’s behavior versus an LLM’s behavior. Programming languages are designed such that you can reason about it. With LLMs it’s always an experiment.
But you're comparing vibe coding to using a compiler. This is an important comparison but you don't need to use LLMs this way; that's why, I think, senior engineers are more effective with LLMs than juniors.
When coding, a few things are happening. You build a representation of what you want to achieve in your head and translate that to code, aka "typing the code", which is actually a pretty complicated process but certainly not all of the entirety of the software engineering process. Then you review what you wrote and commit.
With LLMs you still maintain steps 1 and 3. you reason about the solution, translate it to English and then let the LLM do the "typing the code". Finally you review the output.
The output review is completely deterministic and you have the opportunity to even tweak the LLM's output to match your mental model. The nondeterministic nature of the LLM isn't super relevant because it just changes how much work you do in this step. At this point it's just like standard coding minus the typing. Ultimately it's still an objective relationship with the compiler. It's very much like how tech leads engineer a system through their teams.
Assuming that you truly understand and own every line of the LLM's output, the model is almost working like a macro.
I agree it can feel like working with a team sometimes, but I wouldn't liken it to a macro. That seems much too idealized.
> You can predict which changes in the source code will lead to which exact changes in the behavior of the compiled program
Can you really ?
I think this narrow view on determinism comes up a lot. Essentially any program can be made deterministic over single inputs by fixing all side-inputs. But there is determinism over classes of inputs too, eg. an algorithm given input X deterministically outputs X+1.
I think the wider view is usually what people mean when they talk about nondeterminism in LLMs. Maybe because computers are traditionally so deterministic the wide view is almost taken for granted by programmers.
It's not the narrow view, it's the only actual definition of determinism. Words have meaning. Determinism means "same cause = same effect". If the cause is underspecified and requires interpretation that varies depending on the interpreter, the whole concept is meaningless. Intelligence (human or machine) operates in an underspecified domain, it makes sense to talk about error rate, misinterpretation, alignment, anything really, but not determinism.
yells at cloud