Bytes & Chips

Hybrid search

Until now, Vatuta has only supported semantic search based on dense embeddings. Although this mechanism is quite useful for searches using synonyms or for small queries where little context is included, it is quite inaccurate for matching entities, IDs, or exact terms.

To improve the search for exact terms or entities, sparse vector indexing has been added to the search mechanism. BM25 is a probabilistic lexical ranking function that builds on term-frequency and inverse-document-frequency signals while adding term-frequency saturation and document-length normalization. BM25 prioritizes documents that contain the query terms when, at the same time, those terms are not contained in the rest of the documents. It prioritizes exclusive and unique document terms.

It is expected to improve the robustness and recall of chunks and documents when the search is based on Jira ticket IDs, users’ conversations, or specific contextual terms and concepts.

The ingestion and retrieval processes have also been reviewed, as the processing of the text or content is quite different and more elaborate than the one used for embedding vectors:

1. The tokenizer used in this case is a SimpleTokenizer that splits text by spaces and punctuation marks. Special characters are removed, and the text is converted to lowercase.
2. Stop words such as the, of, an, and with are removed and not indexed by using the FastEmbed library, which supports around 20 languages, including English and Spanish.
3. The Snowball Stemmer algorithm is used to perform stemming on the tokens. Stemming maps some morphological variants to a common approximate stem, reducing the vocabulary size and improving lexical matching. By reducing the tokens to their roots, we reduce the size of the indexes and group the tokens by their roots. Inverted, inversion, and inverting all refer to the same stem, invert, for example.
4. FastEmbed generates one sparse-vector dimension per processed token. For documents, the value is the BM25 term-frequency component, including frequency saturation and length normalization. Qdrant applies the corpus-dependent IDF component at query time.

When searching for the documents or chunks most closely related to a query, BM25 calculates the most relevant documents by ranking them by score. The score is calculated in BM25 for every document in the corpus against the query. The score is the sum, for each query term, of its IDF multiplied by a term-frequency component that applies diminishing returns to repeated occurrences and normalizes the frequency according to the document length. The IDF, or inverse document frequency, is the logarithm of a smoothed ratio between the number of indexed units that do not contain the term and those that do. The IDF ratio gives more importance to terms that are rare and belong specifically to some documents than to terms that are common and do not identify particular chunks or documents.

k1 is a factor that controls frequency saturation, so smaller frequencies have a proportionally bigger impact than larger ones. The more the term appears, the better, but it makes sense to consider the first occurrences more than the last ones.

b is the factor that controls document-length normalization, so the frequency of a term is penalized in longer documents compared to shorter ones. Therefore, an occurrence in a shorter document weighs more than one in a longer document. The length-smoothing term reduces the effect of longer documents containing the terms simply because they are longer.

N is the number of documents or chunks indexed by Qdrant with BM25. n(t) is the number of documents in the corpus in which term t appears. f(t,D) is the frequency, or number of occurrences, of term t in document D. |D| is the length of document D. avgdl is the average document length in BM25, but it is not calculated across the corpus in Qdrant; it is a hyperparameter estimated during the design or optimized during the validation of the system.

When retrieving chunks in a search, the new hybrid mechanism collects them using both the sparse-vector algorithm, BM25, and the dense-vector algorithm, multilingual-e5-small. Both algorithms are independent, and their weights and ranks—the relevance order produced by each one—are unrelated.

The combination of both result sets is performed through Reciprocal Rank Fusion, or RRF. It is based only on the rank of the document, not on its score. Therefore, the documents are combined based on the order in which they appeared in each of the algorithms. There is a smoothing mechanism for reducing the effect of the differences between positions based on K, with a common value of 60. The new score of every document is the sum of the inverse ranks smoothed by the K factor, with a contribution of 0 when the document is not present in a ranking. Qdrant supports other fusion mechanisms, but RRF usually requires less score calibration than score-based fusion because it operates on ranks. Nevertheless, its constant and input weights should be evaluated on a representative query set.

The K and w factors of the RRF mechanism, the k1 and b factors, and the avgdl hyperparameter from BM25 have not been evaluated in any way and have become part of the technical debt of the project.

Semantic embedding model

The embedding model is the model that translates the content, query, or chunk from the sources from text into a vector of a fixed length. The vector is supposed to represent the meaning of the text, so texts with similar meanings should have short distances between them. These vectors are compared with one another using the cosine similarity function in order to compare the semantic similarity of the whole content, instead of term weights like in BM25.

Initially, Vatuta used sentence-transformers/all-MiniLM-L6-v2 as the embedding model. This model is suitable, but it was switched to intfloat/multilingual-e5-small. The reasons for the switch are the following:

• all-MiniLM-L6-v2 is mostly trained using English only, while multilingual-e5-small supports more than 100 languages.
• all-MiniLM-L6-v2 has a maximum length of 256 tokens, while multilingual-e5-small has a maximum length of 512 tokens. The supported chunk size without truncation is twice as large, which allows it to capture a more cohesive meaning, although the optimal chunk size must still be evaluated because larger chunks can mix unrelated information.
• multilingual-e5-small is specifically trained for semantic search and retrieval. It is designed specifically for multilingual retrieval and is expected to outperform all-MiniLM-L6-v2 in this use case.

There was a bug in the original (version 0.1.0) chunking implementation. I did not consider the maximum size, or number of tokens, supported by the embedding model. Therefore, it was possible for the chunk length to be bigger than the length supported by the model, causing the chunk to be truncated to the model limit. This may cause the loss of the meaning contained in the truncated part. To fix this, the chunking algorithm now considers the length limit configured as a parameter, but also the length constraint imposed by the embedding model. It then uses the minimum value of both when splitting the content.

In addition to switching the embedding model, the whole processing of the content was reviewed:

• multilingual-e5-small requires some prefixes to be added to the content before the embedding is calculated. As described in the model card, we must add a prefix to the content depending on whether it is part of the RAG content or the query itself. This is required because the model was trained using these prefixes, and RAG performance decreases significantly if they are not used. The prefixes represent the asymmetry of the content depending on whether it is part of the query or the response. The query content must use the query: prefix, and the RAG content must use the passage: prefix.
• Although cosine similarity is used and it already performs normalization of the embedding vectors, a setting was added to force vector normalization in all cases. Normalization allows the cosine similarity calculation to be reduced to a scalar product. much faster.
• Embedding models do not require stemming, lemmatization, or the removal of stop words. They require the whole text and its word order to preserve the semantics of the sentence. This is contrary to sparse models, whose performance improves when using the former mechanisms.
• The tokenizer used by multilingual-e5-small is SentencePiece, a tokenizer that performs quite well when switching languages or using several languages. Its tokenization uses spaces as part of the tokens, supporting in this way languages that do not use spaces to split sentences into words. multilingual-e5-small is based on the XLM-RoBERTa language model, which uses SentencePiece, but it is specialized for semantic search.

Instrumentation

After the bug explained in the previous section, instrumentation was also improved to obtain more metrics that allow the ingestion and chunking of sources to be monitored:

• ingest_documents_total (Counter): Tracks the total number of high-level document units ingested. A document unit represents a single entity from a source, such as a full Confluence page, a Slack channel thread, or a Jira issue.
• ingest_chunks_total (Counter): Tracks the total number of individual text chunks generated from the ingested documents. It helps measure chunk density.
• ingest_document_size_chars (Histogram): Measures the distribution of document sizes in characters before chunking. It is useful for understanding the size of incoming raw documents.
• ingest_chunk_size_chars (Histogram): Measures the distribution of final chunk sizes in characters. It helps confirm whether the chunk-size strategies are working as expected.
• ingest_chunks_per_document (Histogram): Tracks how many chunks are generated from a single document. It is useful for identifying documents that generate too many or too few chunks.
• ingest_chunk_token_budget_ratio (Histogram): The ratio of the chunk size, in characters, to the maximum character capacity of the embedding model, which defaults to 1024. A ratio (> 1.0) indicates that the chunk is too large and will be silently truncated by the embedding model.
• ingest_embedding_latency_seconds (Histogram): Tracks the time, in seconds, spent generating embeddings, for example during semantic splitting strategies.
• ingest_chunk_split_reason_total (Counter): Tracks the trigger that caused a chunk to split. The reasons differ depending on the source type. The reasons can be:
  • time: The maximum time interval between items was reached in Slack.
  • size_chars: Aggregated chunk character limit reached.
  • size_count: The maximum item, message, or comment count limit was reached.
  • semantic: The cosine similarity between consecutive items dropped below the threshold.

Vulnerabilities management

The version also includes several dependency upgrades to fix the vulnerabilities found after version 0.1.0 was released. However, the diskcache dependency still has a vulnerability because it uses pickle to serialize the cache content to disk. Pickle is considered a threat because it may serialize or deserialize executable content. An alternative is still needed for the persistence of the entity cache.