Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well).
It depends on additional filters and whether you want to use vector search only. The upside of using Faiss would be storing the ID as file metadata and embedding it in the Faiss index. However, if you need any other filters or data, you would need to store it somewhere else.
We also want to share details about how HammerBot works on the backend. Right now, our developers are using OpenAI’s text-davinci-003 model, otherwise known as GPT-3, trained on a custom dataset of our articles. The data is stored in a vector database from Weaviate and the bot is coded primarily with Python, using LangChain, a framework that makes it easy to customize AI output.
When you enter a prompt, the server queries against the dataset that’s stored in Weaviate to get the search results. Those are then sent to the LLM to help it develop a consistent response to your question.
A lot of chatbots will talk to you about anything on Earth. But don’t ask HammerBot for a knitting pattern! It’s designed to have limits on what it will talk about; it focuses on the expertise you can only get from Tom’s Hardware, so it may say it doesn’t know or can’t answer if prompts that fall outside of its training.
What you propose is certainly possible, you could do that with a vector database like Weaviate (https://weaviate.io). Import the data, vectorize it, and connect it to a person with cross-references
Well, there is is still a lot to do. Going from memory to disk, etc. But I see your point.
What might be a bit confusing is the mix of vector search libraries with vector search engines. It's a bit like comparing an inverted index library with SOLR :)
But we (I.e., the wider vector search ecosystem) are working on this
Agreed with your observations about the difference between the libraries and databases. Maybe for a follow-up article? It's nice to see some independent research on the topic, tho.
Yes! We carefully distinguish vector search vs vector databases throughout the article. The end sections on architectures touch on that, such as questioning compute-tier solution vs not.
Reality vs marketing seem to diverge here. On the compute side, Faiss and some others have scaling extensions (GPU, bigger-than-memory, ...) that both out-perform and out-scale some of the vector database solutions. Likewise, similar to redis, teams will use it in a rather persistent way in practice. Conversely, some vector db teams market traditional db notions like CRUD operations, but it is unclear how much that is their ideas talking, the need to meet VC funding stories, niche use cases, or what managed vector search users want. Likewise, if we did a comprehensive benchmark of vector databases vs say graph databases, I'm not sure which would win in many key cases, including on those vector databases advertise on.
Agreed with fzliu, you can also use https://weaviate.io (disclaimer, I'm affiliated with Weaviate). You might also like this article which describes why one might want to use a vector search engine: https://db-engines.com/en/blog_post/87
1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat