I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?
I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?
LLMs do tend to reinvent the wheel. There are off-the-shelf solutions. In-process, the major options seem to be Meta's Faiss, Spotify's Annoy and Alibaba's zvec. zvec's shared libraries are tens of megabytes in size, and I haven't looked closely at the others. My program is in Go, so even C bindings can be a pain.
If you have billions of vectors, you should use a dedicated implementation: nearest-neighbor algorithms in high-dimensional space can be tricky, and there are a lot of trade-offs. My case is amenable to a simple implementation because I don't need huge scale. That's also why I did not need or want an out-of-process server.