Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best.
1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.
2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.
3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:
> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.
They argue against their own point again just after:
> For a biometric pipeline, matching the reference exactly matters more than being faster than it.
Okay, then don't rewrite it! It's not faster anyway!
4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.
Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.
Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.
Then the article should have talked about those things and not pointed at nonsense.
Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos.
For open-source code to get adoption and support, the theory of its construction needs to be understood by humans because at the end of the day, some human will be responsible for its operation. I would call this the Terrence Tao principle: Output of AI is only as useful as how well it can be digested by humans.
I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.
vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.