The purpose of vectors
Why take the trouble and cost of incorporating vector embedding retrieval into an application?
The right answer is: Because it can improve retrieval quality – recall and relevance in information
retrieval terminology.
Vectors can help with this because they introduce semantics into an application: Rather than just
matching by words and exact fields, vectors allow fuzzily matching the meaning of a query to the
meaning of your content.
Simple vector search works badly
While this sounds great, vectors have limitations. The fuzzy matching nature of vector embeddings
make them unsuitable to cases where precision is important – such as finding a particular product,
detailed address, or warehouse shelf.
When applied to textual data, the academic community has shown that even simple bm25 text search
outperforms vector similarity,
and the
industry
has
converged on combining both text and vector search, both in finding candidates and in scoring them (relevance).
And in all applications, methods that use many
more
detailed
vectors
along with tensor math to retrieve them produce vastly superior results to simply representing each
content item by a single vector.
When you are working with vectors you are doing search
Using vectors effectively requires thinking in terms of information retrieval: A variety of conditions
generate candidates, and those are scored (usually in multiple phases to save on compute) to produce
the most relevant, which are then returned to the user or LLM. This process is very different from a
lookup in a database, and is inherent in vector search since all vectors are matching every query,
just with different proximity scores.
Since using just vector proximity does not provide good results, additional factors must be leveraged,
such as multiple detailed vectors, lexical matching, structured fields and so on, and these signals
must be combined into a final score using tensor math and small machine-learned models. Scaling this
process cost effectively to large amounts of data and high query rates leads to a
different architecture
than a database.
This is why search engines and databases are separate product categories even though text (and now vectors)
can be stored in databases: It is not about the storage, it is about the ranking.
Relevance is crucial for RAG applications
It is well understood that relevance is crucial in information retrieval applications for humans,
whether for actual end user text search, or in implicit searches such as for product recommendations.
What is less widely appreciated is that relevance is much more important when retrieving data for an LLM.
This is because contrary to humans, LLMs are not online learners: They do not constantly pick up useful
information from going to meetings, reading their mail and so on. After training they learn nothing and
so are completely dependent on getting all the information they need delivered to them by information
retrieval at the point in time when they need to do some work.
No amount of model intelligence can compensate for missing the information needed to perform a task,
and therefore most of the work in creating RAG applications that reliably delivers quality are in
search relevance: More precise modeling of the data, leveraging and updating more signals and vectors,
applying ever better machine-learned models over larger amounts of the candidates and all the other
tasks familiar from information retrieval.
The tradeoffs between using a database and a search engine
For organizations that already have much of their data in a database, it can be attractive to simply
leverage the vector support added to that database rather than introducing new technology for vector
use cases. This makes for reuse existing integrations, knowledge, operational practices and vendor
relationships.
However, if the purpose of using vectors is to improve quality or unlock new use cases, it is worth
considering that vector retrieval inherently means moving into the category of search, and that
achieving quality goals with vectors takes a lot more than simple retrieval to a single vector:
Most of the work and features needed lie in relevance, and these cannot be architecturally
separated from the database since this means sending too much data over the network, which is why
search engines both store data and perform the relevance functions.
A fruitful resolution of this tradeoff may be to separate into two cases: Where vectors are applied
without a need for achieving particular quality levels, the existing database can be applied to store
them and do simple lookups by proximity, while where quality matters, a proper vector-enabled search
engine which supports the necessary relevance work can be leveraged.
