Won't that be slower than vector DB's by an order of magnitude or more?

I guess the major foucs in certain uses cases is not speed but accuracy and retrieval quality.

Faster is not always better. In certain situations, we may choose to sacrifice speed for increased accuracy.