I would happily jump ship for any competitor that offers solid AI inference benchmarks at a competitive power efficiency, but as far as I can tell Apple owns that market by a pretty big margin. I’m sure someone will point out if I’m wrong.
I would happily jump ship for any competitor that offers solid AI inference benchmarks at a competitive power efficiency, but as far as I can tell Apple owns that market by a pretty big margin. I’m sure someone will point out if I’m wrong.
Ryzen AI are competitive with Macs in absolute and power efficiency terms. There is a Framework 13 with them I believe.
Humans own inference power efficiency by a much bigger margin
Owning a human is much more problematic, and anyway a human’s peak efficiency might be higher, but humans have to sleep and take breaks so overall probably not.
Humans also tend to hallucinate a lot, but it’s not polite to use that word for them. With humans we say they were wrong, because offending a human reduces performance and you can’t reset their contexts.
I just have a desktop at home that I run inference off of. It is a great setup and I don't find myself wanting to inference models directly on my laptop.
That’s what I would do too, but I haven’t found a desktop build that can rival a Mac Mini or Mac Studio on performance per watt. I haven’t looked super hard, but it seems like Mac is in a different ballpark.
I mean, once it's a desktop, watts are pretty cheap, so it's a bit strange to optimize on that factor for the desktop form factor. For laptops it makes a ton of sense.