for example, models running at like 100-150 tokens/second (or faster!) vs 15 t/s
(fable/sol are ~60 t/s, and OpenAI just announced their Cerebras partnership(?) for "ultrafast" mode of 750 t/s)
models aren't able to run that fast right now on our consumer/prosumer hardware. M5 Max for example has a memory bandwidth of 600 GB/s. a 5090 has 3x that, so running the same model on a 5090 is that much faster (provided the model is within 30GB).
running a bigger model on an M5 Ultra is still much slower than running it on a Blackwell chip with sufficient vram, CUDA being a major difference. if apple can bridge this gap, interesting things will happen... and just imagine if M7 Ultra has comparable speeds to Blackwell (or even Rubin)!
There should be some kind of moratorium on new accounts. HN's always had waves of newcomers, but their impact was always limited. The wave passes and people either get filtered out or adapt. That doesn't seem to be happening anymore, since bots can churn out endless gibberish.
He did answer you though. Native is x10 the non-native speed. 50/50 that's not a bot; though it could be a meat-proxy
I am talking about my general impression not this particular occurence. I have a suspicion that someone/some entity is buying old accounts to bypass the new accounts penalty. I even created (https://chromewebstore.google.com/detail/hn-users-filter/ine...) to filter these accounts/comments (disclaimer: vibe-coded)
for example, models running at like 100-150 tokens/second (or faster!) vs 15 t/s
(fable/sol are ~60 t/s, and OpenAI just announced their Cerebras partnership(?) for "ultrafast" mode of 750 t/s)
models aren't able to run that fast right now on our consumer/prosumer hardware. M5 Max for example has a memory bandwidth of 600 GB/s. a 5090 has 3x that, so running the same model on a 5090 is that much faster (provided the model is within 30GB).
running a bigger model on an M5 Ultra is still much slower than running it on a Blackwell chip with sufficient vram, CUDA being a major difference. if apple can bridge this gap, interesting things will happen... and just imagine if M7 Ultra has comparable speeds to Blackwell (or even Rubin)!
Meanwhile the GB300 used by hosted llms:
GPU Memory Bandwidth: 7.1 TB/s Interconnect Bandwidth: 900 GB/s bidirectional
https://pi3g.com/nvidia-gb300-specifications-including-memor...
If you think M7 will hit even 15% of these speeds you're very optimistic.
A hosted instance serves multiple customers at a time. A local model only one.
How many though? At 1m context you quickly fill a full gb300's 280gb of memory
I assume they mean same t/sec as a SOTA cloud model
There should be some kind of moratorium on new accounts. HN's always had waves of newcomers, but their impact was always limited. The wave passes and people either get filtered out or adapt. That doesn't seem to be happening anymore, since bots can churn out endless gibberish.
He did answer you though. Native is x10 the non-native speed. 50/50 that's not a bot; though it could be a meat-proxy
> There should be some kind of moratorium on new accounts.
OC was registered in 2016 though? What do new accounts have to do with this?
I am talking about my general impression not this particular occurence. I have a suspicion that someone/some entity is buying old accounts to bypass the new accounts penalty. I even created (https://chromewebstore.google.com/detail/hn-users-filter/ine...) to filter these accounts/comments (disclaimer: vibe-coded)