In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough better to justify moving away from 3.6 35B even though 35B is still faster. The Unsloth weights don't come with vision support but you can add that back yourself. llama.cpp recipe in case anyone else wants to save some time on setup: https://pastebin.com/fcqsbDTv
> Better... than 3.8 27B
How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.
Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...
Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.
Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.
MOEs are built by training a second "router" model to identify which parts matter inside the dense model.
Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.
Modern LLM's are wildly inefficient.
We should still expect significant performance improvements.
I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.
My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).
https://github.com/rdaum/eider/
It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.
Probably. I've spent zero time with optimization at this point. Code is all new this morning.
Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.
Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
Ok, at 50k context its about 126 prefill, 13 generation.
Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.
Kinda wish I had a Strix Halo here to play with as well.
I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.
Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.
But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
I know a little bit about this problem space from previous work (we were working on performance-portable deep learning back around 2016). The infrastructure has improved but as far as I can tell not many teams have really "squeezed the toothpaste tube" and worked through performance issues systematically. These days a small team and robots can probably do it though.
At my day job I may get access to big AMD AI iron in a couple months (to do research/performance tuning with). That could be interesting. Though that's likely to be of a very different shape from consumer Vulkan. I'd still like to have a Strix Halo to futz with. But I'll wait for RAM prices to drop. (Hah!). I do have an older BC250 board lying around but that only has 16GB RAM.
I got prefill up to 190 tok/sec just now, BTW.