Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.
You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.
Genuinely curious, what is the benefit of doing this over just programming it yourself? It sounds to me like this is just an extra step that will grant you less control?
When programming a feature, you are simultaneously doing at least four things: 1. Implementing. 2. Building highly detailed mental models. 3. Learning and expanding your skillset. 4. Quality control and scope limiting.
And this process can be iterative and dynamic. Writing massive, super detailed specs that you then hand off to a undeterministic model feels like doing step one and three, while skipping two and four, which you then have to do after. What is the benefit? The speed up, in my opinion, comes if you skip step two and four, but then your product WILL be worse. Feels like I am going crazy?
Programming was never the bottleneck for software dev?
> Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual.
I think this lesson is getting partially outdated. Yes, you need to be specific about what you want, and with earlier LLMs, you need to had both domain knowledge and some general software development experience to front-load various big and small choices about design, architecture and operational reality - what libraries to use, how system components communicate, how you handle auth and store secrets, etc. Otherwise the LLM would pull some random mix of ideas from its latent space, and give you something that's broken in really stupid ways.
Nowadays, it doesn't feel like that to me, not anymore. I still need some understanding to verify the proposals, but I found the last ~6 months of SOTA models to make good choices. Like, just yesterday I asked Claude to design me some simple service, and focused on explaining it the domain parts (nature of systems I want to integrate together, the purpose of that, and the user's priorities and use cases), and the design I got back had specific suggestions around security, authentication, deployment, failover, integration, behavioral impedance-matching between integrated systems, and more, that I all recognized as based on solid software engineering and ops practices, but deviating from it explicitly in every place where it would be wrong for this specific project. The model considered way more corner cases than I did, and I'm actually really impressed by it.
But then, I find greenfield development is easy with LLMs. Modifying existing systems, especially legacy ones, is where I need to babysit and micromanage models - because any misunderstanding or inaccuracy, which often comes from stale documentation or naming mistakes, tends to get amplified and confuse the agents. No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
The way I see it: models are highly biased to treat everything they read as "ground truth", all of equal importance. There's no nuanced notion that some information may be stale, that there's a temporal and causal order to sources, and that some information may just be wrong.
And this compounds when you let your LLM write code and documentation over time.
>> No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
True! hence the need for someone to review the final spec output and own it as their own output. I have also found LLM to be better at debugging and solving 'a' specific problem, which I believe is due to output's surface area to be reviewed is lesser in comparison.
But isn’t “grooming” the hardest part? To know the details means to know the codebases behind. So you do discovery, read code, ask people, check db tables, and then once you figure it out, you write a doc with the proposed solution. It gets reviewed and then you code it.
I definitely need AI help for the discovery part… so it always starts with a simple “I need to do X”
You don't need to read the code manually line by line to do that. But yes, you need to know what you're doing.
If you define what you want in typescript by updating existing typescript wordage, you get what you want exactly as specified, with minimal token cost and no ip theft.
I mean then it might be difficult to get stakeholder's alignment on specs written in typescript though.
Yeah, I think as a general direction we need to allow a useful collaboration with the agents on the issue board.
This is btw why Epiq was developed, to keep the board as code, git-backed, distributed (via an event log mechanism), and with the ability to replay the board, to see what agents actually did:
https://ljtn.github.io/epiq
This looks very cool
thanks for sharing, going through your blog about cognitive debt.
Sorry, I might not have been clear, but the tool and blog is by @jolaflow (my brother) and I'm so far mostly involved as a reviewer and early adopter, but glad you like it. I might be biased, but I think the tool provides something really useful for agentic coding and hoping it can grow into a thriving open source project.
> in fact more specific than usual.
At which point you might as well write the code yourself and get a deterministic result faster, better and cheaper.
Not sure about that. Writing detailed specs you deal just with the inherent domain complexity. Writing code you also get accidental complexity that has nothing to do with the business, whuch not only piles on top, but tend to interact vexingly with the detais of the domain.
Where you would have a point is if you'd say we have excellent tooling for wrangling code, but less tooling and tradition to write and manage specs.
I have realized that it's like giving a task to a brilliant coder who has just joined the org and is more excited and eager than usual. Hence the responsibility falls squarely on you to set scope constraints while ensuring only to-the-point features are developed.
> Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual.
Around February you could get away with very vague prompts to Claude. I feel like models have regressed since
Feb/April was peak for code.-
Wasn't this during the period where they had a bunch of bugs around caching and the models were making loads of weird decisions?
I honestly feel like basically nobody knows anything about these models, it's all just vibes (and I'm no different).
Indeed.-
Anthropic broke their models in spring, denied it, gaslighted everyone who said so, and then all but admitted it: https://www.anthropic.com/engineering/april-23-postmortem (basically doing Anthropic things).
> I honestly feel like basically nobody knows anything about these models, it's all just vibes
This, too. Since only providers know what they actually serve, what they change and what limits they impose.
There are some visible degradations though. E.g. Claude-ish.
As for a personal anecdote: around February I created a rather complex quiz web app for myself and friends with multiple question types, sync between screens, multiple media upload types, multiple scoring and timing types, MC inetrface etc. etc. etc. It took me a week or so in the evenings with rather vague prompts to make it.
Now Claude (and Codex) cannot reliably build a much simpler web app even with precise instructions while also maintaining the visual consistency.
But I will agree with you, it's a feeling, not a precise measurement.
I am guessing (guessing) that "great coding model" and "great model for the widest use" (agentic, long-running, unsupervised, knowledge-worker, computer operator) are competing, conflicting endgames.-
Add to that:
- Of course, "labs" (quotes) are incentivized to throw coders under the bus and aim for the biggest possible market.-
- "Sharp, focused, brief, elegant, precise" editing, as would benefit the coding use-case, is actually token-saving, ergo, undesirable.-
The only thing that can stop this, would be the quality and functionality of the codeslop generated by these models to became so low that it actually interferes with the (alledged) recursive self-improvement of models (ie. models start to perform worse/degrade).-
Until such a time, we serfs, will eat what's on our plate, pay for it, and continue to kneel before the machine god overlords. And be glad for it.-
Probably because you were impressed by whatever it produced to fill in the gaps at the time. But if you expect the models to read your mind you're going to have a bad time.
I constantly do various greenfield side projects for myself and friends.
I feel like need much more precise instructions much earlier in the process now than when I was building in February.
Actually this was a real incident around the end of Feb this year when we had just started experimenting with spec driven development (SDD).