Having rolled my own RAG the other week, I personally would recommend talking through it with Claude Opus or a similar model.
My baseline was (vibe coded) full-text search with SQLite, and we landed on long chunks with overlap, with a really simple nearest-neighbor search: quantize the index to a sign bit per scalar, which makes it super cheap to estimate dot products, then calculate better (8-bit quant index times native precision for the query) dot products to sort the top documents. A vector database would make sense for a much larger corpus, but I currently have fewer than two million rows. Claude vibe-coded it to use an OpenAI-speaking local inference server and made semantic search an optional augmentation for the full-text search.
I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?
LLMs do tend to reinvent the wheel. There are off-the-shelf solutions. In-process, the major options seem to be Meta's Faiss, Spotify's Annoy and Alibaba's zvec. zvec's shared libraries are tens of megabytes in size, and I haven't looked closely at the others. My program is in Go, so even C bindings can be a pain.
If you have billions of vectors, you should use a dedicated implementation: nearest-neighbor algorithms in high-dimensional space can be tricky, and there are a lot of trade-offs. My case is amenable to a simple implementation because I don't need huge scale. That's also why I did not need or want an out-of-process server.
Having rolled my own RAG the other week, I personally would recommend talking through it with Claude Opus or a similar model.
My baseline was (vibe coded) full-text search with SQLite, and we landed on long chunks with overlap, with a really simple nearest-neighbor search: quantize the index to a sign bit per scalar, which makes it super cheap to estimate dot products, then calculate better (8-bit quant index times native precision for the query) dot products to sort the top documents. A vector database would make sense for a much larger corpus, but I currently have fewer than two million rows. Claude vibe-coded it to use an OpenAI-speaking local inference server and made semantic search an optional augmentation for the full-text search.
I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?
LLMs do tend to reinvent the wheel. There are off-the-shelf solutions. In-process, the major options seem to be Meta's Faiss, Spotify's Annoy and Alibaba's zvec. zvec's shared libraries are tens of megabytes in size, and I haven't looked closely at the others. My program is in Go, so even C bindings can be a pain.
If you have billions of vectors, you should use a dedicated implementation: nearest-neighbor algorithms in high-dimensional space can be tricky, and there are a lot of trade-offs. My case is amenable to a simple implementation because I don't need huge scale. That's also why I did not need or want an out-of-process server.