You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
This may be platform dependent.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
[0] https://learn.microsoft.com/en-us/visualstudio/code-quality/...
Code analysis can’t detect a sizable fraction of all possible errors because for any Turing complete language the number of possible errors is infinite.
I don't think I implied that they could.
Each of these are just layers of control at different lifecycles of agent code generation. Analyzers are nice because it gives targeted, static analysis that can prevent certain classes of errors very early and at lower iterative cost (e.g. a build)
I said
“You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.”
And you replied
“This may be platform dependent.”
I’m unsure how else to read that other than an implication that this might be possible on some platforms.
I would certainly consider Roslyn analyzers capable of covering "a tiny fraction" of possible errors :) They are quite capable of covering for many common types of structural coding mistakes.
I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.
I've been using languages with stronger type systems and that's also a huge boon.
> better code than a human would write,
Why would that be the case? You can run human written code through the same “linters, compilers, static analysis, fuzzing, testing” as you do agent produced code.
You would be surprised. Humans will do human things like be extremely inconsistent, ignore warnings (if they are not enforced as errors), skip steps because they are lazy (devs often chose to skip our pre-push hooks and preferred to run in CI and babysit the PR).
Agents can also do all of those things, but they are generally more compliant to instruction.
That’s an assertion with no evidence that doesn’t match my experience.
Agents require far stricter guardrails than humans. Without linters, tests, static analysis, oracles etc… no agent can create a large program.
Even if you’re correct, you just build those checks into CI so that neither humans nor agents can skip them.