Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”.
Dumbed down quantization?
No. Full intended inference weights preserved, so far so good.
Slow performance?
No again. Looks like you could get over 150 tokens/second.
Give up context window size?
Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this range, and quality does start to drop off toward the full size.
In my experience the 1M context is genuinely too much. The first time I swapped from OAI to DSv4P, I checked and double checked that the harness/etc was working correctly over the course of hours and hours of work thinking that I had set something up wrong because it simply never had to compact! The drop in quality is arguably less than that of what you get from compact to compact on Codex, which is good for what it is or was.
Was also surprised to learn just how much of Codex's window was being burnt on shit I didn't want or use. Sure I can pass this and that flag to eliminate most of it, but for a $200/mo product aimed at professionals, that isn't something anyone should have to janitor (also totally ignoring the bandaid of banked resets they've slapped over their repeated mistakes).
It's wild just how far $20 will get you with Deepseek, even at their new rates. Buyers Remorse is my very least favorite feeling, I felt sick thinking about what the $1200 I had given OAI this year would have gotten me had I only tried sooner.
I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.
Experts aren't trained on separate tasks. More recent routers are designed to spread out requests even more evenly, and they were already pretty even.
Yes. It's called REAP and from what I've seen, results aren't stellar.
Glazing over a lot, that's how they work already, just not in the way you think. A relatively small fraction of the model is active at any given time