No benchmarks, no info on which models are used, ai generated video, just a signup page with nothing else.
Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."
The evaluator is public here: https://echo.tracerml.ai/eval/
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
New video is also being made.
> No benchmarks, no info on which models are used, […]
The benchmarks are here → https://echo.tracerml.ai/eval/
They are not good benchmarks but at least they exist.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks.
In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
It's basically trying to replicate OpenRouter, which works pretty well and has a lot of nice features to abstract away any single provider, such as failover, metering, autoswitching, etc. It's actually a really smart infrastructure abstraction.
I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"
Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.
It feels a bit more like Fugu to me, which acts as a multi-LLM orchestrator (though I think Fugu combines open- and closed-weight models), but without being able to see the “secret sauce” behind how any of them decides the number of "plies" each model in the swarm gets, they all feel rather difficult to compare beyond the big public benchmarks...
https://github.com/SakanaAI/fugu
OpenRouter's model router for coding isn't really sophisticated. This is very different.
Unrelated - but reminds me of my favorite quote by tenderlove:
"microservices turn function calls into distributed computing problems"
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug
It's easy enough to copy and paste in a prompt, no?
Eval tests while giving general indicators might not be similar for each use case.