> They do not beat opus on real-world usage
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
[dead]
[dead]
>> We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios
Okay but the parent said real-world usage, presumably meaning coding tasks.
We have a whole bunch of complex evals that Haiku 4.5 passes. That doesn't mean it is a good model for coding.
Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
0% is not 4% less than 4%, that would be 3.84%.
0% is 4 percentage points (pp) less than 4%.
[dead]