I tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.
I tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.
are you evaluating using OpenRouter.ai or other routers/gateways?
they have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.
What's your use case?
I bet it is large-scale data crunching for labeling or categorization.
Probably not because this is usually not a time sensitive task.
And the fact that you will never be benefited from the batching they’ve explained either, you’ll be paying for every token sequential or parallel unlike nvidia or amd gpus