My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
Anyway, here's mine, still wip:
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I think that dissolves the "self driving" distinction, though? The mechanisms for driving the codebase must be present in the codebase itself. Otherwise its just a regular out-of-band development process.
fine-tuning may be a more scalable approach to LLM personalization than sending all the same context to two LLMs
I'm working towards both in my homelab to see which works better with little qwen
[dead]
I’ve definitely had a hunch about a software factory harness kind of thing.
Your thoughts on the matter seem clearer than mine.
I’m kind of relieved to see so many stages; that’s something I’ve grappled with conceptually. The volume of (sometimes optional) stages has made it hard to resist the temptation to hack something together with a GHA workflow.
I salute your choice to support distributed; that’s a requirement in my book.
I look forward to trying it out.
Expressing complex emergent workflows is fairly straightforward given how the workflow engine recurses:
- a workflow is a series of steps
- a step is one or more tasks that can be done in parallel
- a step is complete when all of its tasks are complete
- a task can be a unit of work, or a pointer to another workflow