You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
Check out eugr’s TP=1 sparkrun recipe :)
It’s an NVFP4 quant, but it fits, and is surprisingly capable.
do you have a HF link? HF search is not uncovering it for me
(or is it somewhere else)
https://github.com/spark-arena/eugr-recipes/blob/main/recipe...
This one!
I'd recommend pointing your agent at it (after installing sparkrun), and asking it to research the absolute latest in TP=1 Flash-Next - mine grabbed particular vLLM nightlies and mods to improve performance, and it was well worth it.
I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me. I'm not keen to run nightly vLLM, too many issues with it in the past. Going the qwen-next path means displacing things I use daily :/
I have a watchful eye on the diffusion ~ Jev/Kev PR
https://github.com/vllm-project/vllm/pull/57250
For what it's worth, Flash Next outperforms every other model that is available to us on the GB10 in all of my testing; though if you have two sparks then the TP=2 version is even better and easier (I don't think you'll need the nightly for that at all, just use the recipe)
I'm so tempted to buy a second one...
prices have gone up quite a bit...
I'm running embedding, reranking, and policy tuned models too, and a Jev/Kev when that's landed. Flash Next is not a substitute for those
I have OpenCode/Fireworks to access big models
looks like this is likely it
https://github.com/spark-arena/eugr-recipes
https://github.com/eugr/spark-vllm-docker