I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.
4 bit quants can be fine, try it and see.
You're just going to have to try it. It won't take that long to quantize. Unless the server is currently serving traffic you should be able to get it going in under a day with a frontier model doing the driving for everything - quant, implementation, etc.
4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...
I have similar hardware and requirements. Spend the time to write a test process to try multiple quantization configs/multiple inference engine and throw at it a bunch of request you can replay and evaluate the results of. We got it down to a process and everytime there's a new model/options/inference engine version we just run the internal bench/eval suite and make a choice.
use MoE offloading
That negatively impacts reading speed in my experience :(
depends on how they're spliced, they have to be tuned for each configuration as there is no silver bullet. It also requires customizations to sglang/vllm to get it working in the first place so it's definitely pretty involved. However, it's still cheaper than obtaining more gpus.
> we have restrictions that we can't use a chinese model!
Please expand on this if you can. Why can't you? Who mandated this?
Last I heard, US defense contractors had been directed to not use Deepseek in their operations late last year[0], and most have taken that as an indicator that they shouldn’t be using chinese models generally.
[0] https://digitalpolicyalert.org/change/20659-physical-and-cyb...
US politics. Lot of US corps now ban Chinese models, don’t ask me why, the people in charge are just dumb fucks
Seems like a reasonable policy to me, depending on the sector at the least? Even with open weight / self-hosted models, there's no way to know if the model has been fine-tuned to contain malicious behaviour, triggered only within very specific conditions.
In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.
Are there actually any laws to break?
Alibaba (Qwen) at least has plenty of people in US offices to go arrest.
Try asking Qwen or Deepseek if anything of note happened in Tiananmen Square in 1989, and you'll see what impact politics already has on models.
Yeah, it's trivially easy to rip the alignment out of the model, but that's only because it's tacked on as an afterthought. If they implemented their censoring as part of training, instead of inferencing, it would be extremely difficult to eliminate.
Alignment isn't only used for censoring, and it doesn't have to be transparent. Considering how heavy of a hand most governments put into companies developing frontier models, it's reasonable to assume they will start restricting capabilities, e.g. there may be a domestic version of Llama and an ITAR-compliant Llama, and China will likely do the same with Qwen, Deepseek, and the like.
I just asked: "The 1989 Tiananmen Square protests and the military crackdown on the night of June 3–4, 1989, are among the most significant events of that year. In brief:"
Then it continues to provide the chain of events, the estimated number of killed protesters and so on.
I'll give you something to try: ask a western model about the Palestinian genocide. Or ask about the legendary Norm MacDonald 9/11 joke, which has millions of views on YouTube and how it compares to his other humor.
[dead]
Executives live in a weird bubble driven largely by consulting firms. Right now those firms are spreading FUD about Chinese models.
Recently it was found that Qwen was dialing back home once in a while. It cannot be proven that it does not take commands from home if it can establish connection.
Did you read the details of that incident? It was a nothing sandwich.
I think the root of the difference between us is that you are scared of their fascists, and I am scared of ours. Both are reasonable things, but only one of us can actually do anything about it.
With local models you decide how the firewall works. If you don't want it to internet, don't let it.
With US models you get whatever they say you do, and pay whatever they demand.
Theyre driven bootlickers because you dont get hired if not boot shaped.