> and spent $165,000 of tokens on the most advanced coding LLM anyone has access to.
After having used 2 full weeks of 20x Max plan tokens on Fable over the weekend (coding all day Saturday and Sunday on a non-trivial project, tasks across full stack, mix of adding features, reviewing code, and fixing bugs), I’m confident if he’d spent $165,000 in Opus tokens the port would have gone more or less just as well (and probably for less than $165,000). Especially so with the system they set up with all the custom workflows, adversarial reviews, extensive test coverage, etc.
But I get your point is probably more about Jarred’s experience level and the high cost than the specific model used other than it being SOTA. I’m just being pedantic and feeling a bit disappointed with Fable’s real world performance after all the hype.
> I expect if he'd spent $165,000 running Fable against the Zig version he could have got a 5% performance improvement, too.
Totally agree and in fact I’m sure it could be done with significantly less cost even if they stuck with Fable instead of Opus which I’m sure could also do it.
Fable is kind of fantastic on the difficult tasks. if it's something eithe rmodel can do then you can't see the difference. Fable also makes much less mistakes. It's a more relentless, proactive problem solver.
I’ve been noticing that it will test hypotheses for evidence frequently. I really like that. With Opus I have to instruct it with a skill to build prototypes, but Fable will do it and do it really well by default. Really nice detail. I’m guessing it’s part of the system prompt, but the higher capability paired with this experimentation lean makes it far better at planning and verifying.