Chinese labs are so used to manipulating benchmarks to try to flatter inferior models that when they finally have one that's really pretty good I think the official announcement here undersells it.
That's pretty solid. Smarter and cheaper than Luna xhigh, not as smart but less expensive than Luna max. Smashes deepseek v4 flash, and even worse it matches v4 pro at a tiny fraction the cost. Roughly equivalent to sol medium, at a fraction the cost.
They should've just lead with real, up to date data, because it's good, not the silly old tactics like comparing to Opus 4.8 when 5.0 is out in many of their charts.
Congrats to them!
Maybe others have found otherwise, but I find the benchmarks drastically different to real world "feel" of a model, even within the same harness. I'm not sure if this just reflects personal interaction styles, or if it is indicative of benchmaxxing or unrealistic automated benchmarking methodology.
Opus 5 consistently comes at or near the top, but outputs constant unreadable jibberish. Meanwhile GPT 5.6 Luna medium tends to be rated pretty poor on agentic tasks compared to the Chinese lab open models, but I find the latter much more likely to lose track of their own behaviour during a long-horizon task or get stuck in a doom loop.
(This isn't a comment on GLM-5.3 Flash as I've not used it!)
Opus is a yap god. I've found it much, much better with Claude Code's output style set to `Concise` and this:
https://news.ycombinator.com/item?id=49413456
We shouldn't have to resort to this, but it can be mitigated enough that it stays as my daily worker agent. Although, I mostly use Fable to farm out to Opus agents so I don't have as much exposure to what kind of blathering is going on in there.
It's crazy that sometimes I ask Opus 5 to explain what it just wrote to me, and it declares "that was word salad" (its words, not mine, without any hint from me other than "explain it").
I've been using 5.3 since they initially announced it and my gut feel is that it's not as good as 5.2 for agentic tasks. I'm still using it - I don't think it's bad. I'm just not convinced it's better.
idk i think that i spend significant tokens with both to be able to tell 5.3 is way better overall.
https://kommodo.ai/i/IFSUUQYT522uZXWvePZ1
it still fells stupid sometimes and it is benchmaxxed for sure. but its good enough that im building all the hobby projects with it.
I have had the opposite where I felt 5.3 as a stronger model than 5.2. Its feels way more in tune with my code, and does more nuanced edits. Though I do handhold my models a lot, so might fall outside the agentic term.
I don't know how anyone can actually use Luna max on ANY real workload. I've had Sol orchestrate a bunch of Luna agents, these agents were explicitly given small chunks of larger objectives and they still filled their entire context windows with just reasoning tokens, until compaction hit, and then reasoning again.
I've probably wasted a good 40% of my weekly usage on Luna Max agents just thinking and not writing a single line of code.
Are you sure you actually spun up Luna sub agents? Sol up until recently could only spin up Sol and Terra agents and would even name them "Luna" despite not being so, you can check in your usage whether it was Luna or not. I believe now it's fixed though.
https://www.reddit.com/r/codex/comments/1vj3hhn/wait_so_sol_...
https://www.reddit.com/r/codex/comments/1vp0rig/sol_can_fina...
The one time I tried asking Sol to use subagents for a small project, it took a surprisingly long time, used up the entire usage limit in one go, and basically failed the project.
I’m pretty sure that plain Sol, serially, could have finished the task faster, cheaper, and far more accurately. I’m also pretty sure that any competent subagent orchestration could have gotten it done with even very simple subagents quickly and cheaply.
(Is it really that hard to set up a handful of subagents that all use the same initial context and to load that context with what actually matters? The APIs certainly support it.)
I only use Luna (max), I find it very rarely just reasons. In fact, I find it reasons too little.
I've had the same observation that Luna will quickly fill up its context window with reasoning, but it surprisingly hasn't been a problem really.
It will cycle through like 3 /compacts, complete the complex goal successfully, and cost me like 1% of my weekly usage on the $20 plan.
Edit: this is me using Luna directly, not Sol as the taskmaster
Luna max is all I use. In my experience, it works really well for overnight tasks.
I feel the same. I just use it for planning and chatting. Not real coding work.
This is why I stop at xhigh.
If your code is complex enough for Luna Max to fail maybe you need to write a bit yourself so they can copy your idea
It's also better than Sol (at whatever effort) at designing pretty UIs. I have a Codex sub and I've been using this model for UI stuff.
> I've been using this model for UI stuff.
The flash one?
Yes. I have no UI experience, and wanted a model that could produce something good without me telling it how anything should look like.
My prompt was something like: "here's data I have, here's what matters to me, create HTML mockup".
All GPT 5.6 models were laughably bad. And I don't want to downplay it - they were just absolutely, objectively horrible. Every single attempt was what I could probably call "if json was ui".
Claude models produced... "claude look".
GLM 5.3 - somewhere between GPT and Claude.
Kimi k3 - each attempt produced beautiful UIs. It used components that I didn't even know existed and wouldn't even know to ask for. But expensive, very expensive.
ox-alpha (GLM 5.3 flash) was very close to K3. And at this price point, it's already configured as "designer" model in my oh-my-pi.
Yeah, exact same for me. K3 used to be my go to for UI but it's quite expensive. Ox alpha being so cheap and so comparatively good at design is crazy.
I haven't tried it but I think Qwen Max is also very good at design.
Yeah, when it was secretly called Ox Alpha.
Only 73K output tokens too. Anthropic should really be embarrassed with their Sonnet 5 price/performance.
Opus 5 is better than Fable in this benchmark?
Even Artificial Analysis has Opus 5 better than Fable in their aggregated "Intelligence Index" which combines 9 benchmarks. Opus 5 is heavily benchmaxxed.
Depends on the benchmark but yes. I think Opus is more heavily optimized for coding. On the usability side, its output is almost intolerable to read. It seems to code fairly well. Fable is more enjoyable to use for planning/interacting with
> They should've just lead with real, up to date data, because it's good, not the silly old tactics like comparing to Opus 4.8 when 5.0 is out in many of their charts
It's what people know. Opus is just the common target.
> Smarter and cheaper than Luna xhigh, not as smart but less expensive than Luna max. Smashes deepseek v4 flash
The problem with this and DeepSWE is it goes for a very specific profile. I'm not convinced DeepSWE is any accurate in actual work. It's surely a different signal (compared to some that allow cheating) but it has its own issues, e.g. weak harness.
Luna is great at following instructions but bad instructions or anything not covered = death.
Deepseek is more analytical. Good for bug tracking.
GLM is a better all rounder in some ways. Better at creativity.
> I'm not convinced DeepSWE is any accurate in actual work.
They listed Muse Spark 1.2 around DeepSeek V4 Flash even though it's a much shittier model in basically every aspect.
> GLM is a better all rounder in some ways. Better at creativity.
I agree with the creativity part.