Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
How was qwen3.6 launched?
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.