RDNA4 is pretty similar to CDNA4, yet over a year after the release of "pro AI" cards like the r9700, they had basic kernels lacking in vllm (like w4a16 int4 kernels) while they were implemented in the datacenter CDNA4 cards.
AMD hardware runs well on llama.cpp because basically anything runs on llama.cpp, especially with vulkan. It's not high praise of AMD's software team to say llama.cpp runs well on their hardware
Yes, it runs in vllm happily. It gets >900TPS output reliably on a single 5090 with the nvfp4 model.
It's clearly worse than vanilla 26B-A4B, and lacks some things like structured outputs, and gets some tool calls wrong.
So you have to find a usecase or a hand rolled harness that leverages the cerebras-level TPS while not going off track during (even short) tasks.
And it fails on rocm of course. This engine is such a hassle on AMD.
That's AMD's fault.
RDNA4 is pretty similar to CDNA4, yet over a year after the release of "pro AI" cards like the r9700, they had basic kernels lacking in vllm (like w4a16 int4 kernels) while they were implemented in the datacenter CDNA4 cards.
AMD hardware runs well on llama.cpp because basically anything runs on llama.cpp, especially with vulkan. It's not high praise of AMD's software team to say llama.cpp runs well on their hardware