Wow!
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
Databricks wouldn't be using a subscription plan though due to the size of their engineering team which is what matters here. Free lunch is over for basically all providers' biz coding plans
True, but OpenAI are OK with using their subscription plans with Pi, so GPT 5.6 with Pi may be a good combination.
There seems to be a lot of good buzz about GPT 5.6 on Twitter - people (incl. OpenCode team) preferring it to Fable 5.
Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
It seems like they are actually using the subscription providers' respective cli tools and managing context for them. In which case I believe it is not against the ToS any more than invoking codex cli from a custom python script would be? I personally cannot find an openai-hosted clarification, just on hn and x.
Awesome - that's what I do in my own harness for agy. Hopefully someone yells if they get banned.
Obligatory yes, but only if you're subscription-based and not pay-per-token as enterprise users are.