Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
> --cache-type-k q8_0 --cache-type-v q8_0
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
Yes to both.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.
I wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
https://arxiv.org/abs/2512.11280
"AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference"
Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
Deepseek's DSpark does dynamically adjust speculated token count per user/completion.
https://arxiv.org/abs/2607.05147
But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.
Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
Hm, I have a 4090 as well, and:
$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory
Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
There has to be a better way of sharing config , a profile of sorts you can upload and share
Yes, llama.cpp supports presets (an ini format). Though it's not always clear which llama-server flags it does not support.
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.
But maybe someone else has experience to share there
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
You sound like your trying to reassure yourself of something.
I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
> I sure hope my boss doesn't think he built my work!
Most managers do though?
I generally agree and expect this to be the case from a legal perspective.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
[0] https://en.wikipedia.org/wiki/Structure,_sequence_and_organi...
[1] https://en.wikipedia.org/wiki/Abstraction–filtration–compari...
Some people will now argue it was the chisel—not Michelangelo—who created David.
The idea guys will really think themselves gods, now.
No, it's the difference between management and direct work.
None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.
"Carve me a naked guy. Make no mistakes."
I cannot fulfill this request. I am programmed to follow safety guidelines that prohibit the generation of sexually explicit content.
So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?
Absurd logic. The AI built the website.
I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.
I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.
I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.
Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
That’s an interesting way to look at it, and still generous to the llm.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
It all feels a little like that, again.
The director receives accreditation for directing the film, not creating it.
I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.
I'm not sure that typing messages to a chat bot requires much catching up, but since you think this is what I should do rob, I better listen!
If you don't get reliable results, but others do, then maybe you should indeed. But your choice.
You know how many pieces of art Damien Hurst creates himself Vs his studio assistants creating them under his direction?
For example, of his 1500 spot paintings, he only actually made 5 of them.
It's not uncommon at all for artists to work this way.
He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.
There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.
no, because there is another human involved.
llms are not human.
Any chance I could run it on a GeForce RTX 5060 Ti 16 GB, (64 gb ram)
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop?
If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.
How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
For reference, here are also some numbers from my 4060ti + 3060 (16GB + 12GB) setup. [0][0]: https://news.ycombinator.com/item?id=48700091
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Here's my config: https://gitea.va.reichard.io/evan/nix/src/branch/master/modu...
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
Good point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
What makes you say that it would be hard to do that?
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Sure, but front ends like LMStudio exist for that crowd
Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are.
It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
> Why would they wish to handcraft this ?
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The people OP mentioned about "just want AI" .
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
> More and more people are thinking agent native so this is not really a issue.
Why this headache inducing lingo tho? What does that even mean, and why should I sign up for your webinar about that?
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
If you do not want to tweak parameters you can just run
That's itStart by copying the command line from the Unsloth guides.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
https://ollama.com/library/qwen3
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”?
You could override, obviously.
Yes, llama.cpp does that.
just tell claude/codex "set this up on my system $huggingfacelink"
> LM Studio doesn't work behind proxies.
Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet?
I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
> I feel like that's a rather niche situation you got there
Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
> Every big company in the world uses a network proxy.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic).
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
Every big company? YMMV, but I'd say about 20-40% do.
If you're on Linux you can probably use proxychains.
You know free LLMs can help you understand that command line or design your own...
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
vibe sysadmining?