Related thing is that LLMs are still notoriously bad at writing instructions for themselves (so managing themselves also). They tend to overspecify. If I had a copy of myself I would tell him "do the thing", but agents say "do the thing like this and this, it must be such and such, use X and put it together that way". The subagent is just as intelligent but now it's constrained, sometimes as you start developing something you discover some things are better organized in a different way etc. So it spends much time figuring out how to deliver exactly what was asked even if it's not optimal because that's the task.
Basically a company where you can't ever question decisions of the higher ups, not even ask for clarification, but they force implementation choices on you. I mean, it definitely does happen, but the results are usually not great. Not alawys. For some tasks it works.
> Related thing is that LLMs are still notoriously bad at writing instructions for themselves (so managing themselves also).
Isn't this odd? This seems to be the #1 task that the major labs should be optimizing, since it makes agent so much more effective.
> They tend to overspecify. If I had a copy of myself I would tell him "do the thing", but agents say "do the thing like this and this, it must be such and such, use X and put it together that way". The subagent is just as intelligent but now it's constrained, sometimes as you start developing something you discover some things are better organized in a different way etc. So it spends much time figuring out how to deliver exactly what was asked even if it's not optimal because that's the task.
Isn't this just the planning pattern, common for agentic work? (first plan then implement.. with things like /plan, or a planning skill, with or without creating a PLAN.md). If this methodology is wrong when an agent prompt another agent, it's also wrong when the human runs /plan manually..
And Indeed, I always thought it was odd to constrain the implementing agent like that, because common sense is that we discover things as we implement. Perhaps we should prompt the agent to communicate back, as they implement, any shortcomings in the plan itself - such as things they learned while implementing that contradicts the plan.
Things are moving at a pretty much breakneck pace, I'm not surprised there are so many opportunities for improvement yet to be grabbed
Anthropic consistently claims the opposite since Opus 3 (early 2024) which was notoriously better than their modern models at understanding overarching intent. That was literally their official prompting advice, roughly "in our tests Opus-written prompts were better than prompts written by humans, so you should let Opus write them for you". But it was painfully obvious that the model lacked depth to do that, and it's obvious modern models lack it too. It will probably not be solved until reaching AGI, whatever that word means.
AI shops rely on scale automation, metrics, and LLM judges, which heavily distorts their visibility into what they're doing. This often results in being out of touch. That's likely how Claude lost its writing ability (they stopped reading its outputs).
> until reaching AGI, whatever that word means
Maybe that’s exactly it - AGI is ability to plan well?
There are many wildly different definitions of AGI to begin with, but sure, "deep" cognition would be able to accomplish a ton more than the current "wide".
> The subagent is just as intelligent
Isn't it a common practice to use cheaper models to do the grunt work?
Does this actually work for anybody? In my experience, when you use cheaper models as subagents, the more expensive parent agent ends up reading all of the subagent's work and surrounding code into context anyway to review it. So you end up paying more, not less.
Plus the subagent's work is often subpar and the parent ends up requesting multiple revisions. Each back and forth burn unnecessary tokens as well as my time.
Where it might work well is for separate task categories like image generation - get the parent to ask for an image from another model that is better at this aspect
When I am making agents do a hard thing, I use Opus as both parent and child. I make the child review its code extensively, and then still make the parent read the work and review once again. It is much faster and cheaper than just rolling with no extra reviews, and then in a day I have fifty subtle bugs.
Exactly, and a lot of times when I step back and discuss alternative approaches and architectures with the LLMs, they keep going back to say: "the code does not use this (currently), therefore this is fine to throw away", or "the code does this (currently), so this approach is not compatible" They are bad at stopping to ask "what should the code be doing"? It's like they love being constrained by current choices, even if there is nothing unchangeable about those.
> being constrained by current choices
The most obvious reason for this seems to me to be that the data they were trained on likely overrepresented cases where the current choices were treated as unchangeable.
Actual case studies where they talk about what approaches were abandoned and why certainly exist, but I'd guess are much less common than posts presenting a problem as "these are my constraints, how do I meet them?".
I've noticed this myself and am not sure how to solve it yet. I think there is value in methodically specifying what needs to be done ("grill me"), but sometimes the resulting plan indeed becomes overly prescriptive, leading agents to chase goals that ultimately don't matter.
I don't know if my requirements are lax but everytime I give opus 5.5 a job and tell them "spin up sonnet or haiku agents for grunt work they should do given their skills then double check their work"
I get great results and save tokens
[flagged]