At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?
At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?
For a fairly weak comparison, modern Bitcoin mining chips are around 13J/TH, so around 0.32TH per token, or 320 billion double-sha256 mining operations per token served by this system. Peanuts are also about 4200 Joules, or almost exactly 1000 tokens per nut assuming you could efficiently turn it into electricity to run this chip. Just as a frame of reference, I don't think this actually serves to make any point.
Each Nvidia rack is 120 kilowatts, so the power consumption is about the same as 10 racks of GPUs, FWIW.