The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself. This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.
Reminds me of the METR blog post on the HF attach by OpenAI. At some point the agents believed a false fact (that the evaluator would try to figure out if they have cheated on a task) and spent a lot of time trying to find workarounds. At no point did any one of the agents try to verify that fact even though the information was available to them if they looked for it
Yup and this propagates those clumsy if this_new_code_branch: actual_code_that_matters else: old_legacy_code_that_should_not_be_there
> Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.
I was ignoring that, but it did seem somewhat intentional by the human running it? There's a lot of "I'm going to use AI to make a better AI-using machine" projects about that aren't really focused on wider application.
> The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself.
In my experience, it's even worse than that: the LLMs constantly assume that all existing code was created entirely by me.
They generate code, then suddenly start talking about that exact same code as if I had manually and deliberately written all of it. They assume every single technical decision was made by me. They don't just treat it as gospel, they assume it's my gospel.
It's surreal.
> This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.
I did not know this! Any link to an announcement or autopsy of sorts (even if not by the initiator of that project)?
I mean, it was pretty expensive, wasn't it? A few tens of thousands of dollars, IIRC?
This is the closest I could find to a post mortem from the creator:
https://yegge.ai/essays/the-shape-of-things-to-come/
But the GasTown part is barely a single paragraph that I could not make sense of. Like, what’s the Opus “tic”? Why was it so fatal to GasTown? As someone who only ever accessed Anthropic models through other harnesses like Copilot, I have no idea.
I do think what he’s saying roughly resembles what I’m forecasting will be a likely future of software engineering: that it will evolve into crafting comprehensive, bespoke automated validation mechanisms which let you establish high confidence in the agents’ work without really having to look at it.