Say a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.
Say a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.
>I realize the question assumes a major simplification that everyone's prompts/sessions are the same.
well, exactly.
that's tough to answer without just average sampling because some users will ask the model "what's todays date" or "what color is the sky?" and some users will ask "Let's rewrite the linux kernel in brainfuck."
So maybe the better question would be how many concurrent actively-thinking/working agents it could handle?