the only reason they are fast is because the models they host are severely quantized so i've heard.
Huh. I heard a podcast with the founder talking about their custom hardware, but quantization would explain it.
Quantization alone does not explain it. It's mostly custom hardware[0].
[0] https://groq.com/the-groq-lpu-explained/
Why repeat this nonsense when it’s so trivial to just check. The reason Groq is fast is because they employ absolutely ludicrous amounts of SRAM. (Which is 10 times faster than the fastest VRAM.)
they responded to my tweet last year and said they didn't quantize the models.
It's very hard to find right now but I'm sure they said they don't quantize KV cache, but their weights are in fp8.
Huh. I heard a podcast with the founder talking about their custom hardware, but quantization would explain it.
Quantization alone does not explain it. It's mostly custom hardware[0].
[0] https://groq.com/the-groq-lpu-explained/
Why repeat this nonsense when it’s so trivial to just check. The reason Groq is fast is because they employ absolutely ludicrous amounts of SRAM. (Which is 10 times faster than the fastest VRAM.)
they responded to my tweet last year and said they didn't quantize the models.
It's very hard to find right now but I'm sure they said they don't quantize KV cache, but their weights are in fp8.