Qdrant/bm25
The lexical BM25 sparse head, hashed rather than learned, with no neural network, no device and no dtype anywhere in it.
Qdrant/bm25 is the one sparse head with no neural network: it splits, filters, stems and hashes the text and weights each stem by term-frequency saturation. It has no checkpoint, vocabulary, device or dtype, and its output is a SparseEmbedding, scored as on the sparse output page.
Using it
//! `Qdrant/bm25` has no device, no dtype and no checkpoint to fetch, so `build()` cannot fail.
use candding::Bm25TextEmbedding;
fn main() {
let model = Bm25TextEmbedding::builder().build();
let passage = model.passage_embed(&["The quick brown fox jumps over the lazy dog."]);
let query = model.query_embed(&["a quick fox"]);
println!("indices in the passage row: {}", passage[0].len());
println!("query . passage = {:.4}", query[0].dot(&passage[0]));
}| Item | Bm25TextEmbedding |
|---|---|
| Builder | Bm25TextEmbedding::builder(), with no settings |
.build() | returns Bm25TextEmbedding, never a Result, since nothing touches a filesystem, a network or a device; the stopword list is compiled into the crate |
passage_embed, query_embed, embed | take &[S: AsRef<str>] and return Vec<SparseEmbedding>, with no Result and no batch size; embed is the passage side |
.descriptor() | a SparseDescriptor with vocab_size 0, default_max_length usize::MAX, weights NoWeights and computation Computed |
SparseTextEmbedding::builder("Qdrant/bm25") refuses before any fetch, and its error names Bm25TextEmbedding::builder().
From text to row
| Step | Rule |
|---|---|
| Clean | every character that is neither a Unicode word character nor whitespace is removed, and the text is lowercased |
| Split | on whitespace; a script with no spaces between words, such as Japanese, becomes one token per unbroken run, as in the reference |
| Filter | a token that is bare Unicode punctuation, an English stopword or longer than 40 characters is dropped |
| Stem | the Snowball English stemmer; a token whose stem is empty is dropped |
| Index | the absolute value of the signed 32-bit murmur3 hash of the stem, seed 0, as Python's mmh3.hash computes it |
| Passage weight | count·(k+1) / (count + k·(1 - b + b·doc_len/avg_len)), with doc_len the number of surviving stems in the text |
| Query weight | 1.0 for each unique stem |
| Constant | Value |
|---|---|
k | 1.2 |
b | 0.75 |
avg_len | 256 |
A row is as wide as its distinct stems, anywhere in a 31-bit range, and the hash has no inverse. Two stems that collide on one index keep the larger weight.
Limits
- Stemming and stopwords are English only, and there is no
.language(..)method; the reference takes one of eighteen languages. Splitting and the punctuation filter work on any script. - A row's weights are the term-frequency half of BM25 only. The inverse-document-frequency half comes from the vector database's index-time modifier, so
query.dot(&passage)between two rows is not the complete BM25 score. - A word shared with
Qdrant/minicoil-v1does not land on the same index: miniCOIL hashes only the words its vocabulary lacks, and reduces the same murmur value modulo a range before adding a shift.
How it is checked
There is no checkpoint to compare, so a pinned unit-test suite in the default cargo test --workspace checks it against a hand-reconstructed run of the reference pipeline over the nine fixture inputs, plus cases for the punctuation filter, the length cap and stemming.
XLM-RoBERTa (flash)
The XLM-RoBERTa (flash) encoder behind jina-colbert-v2 and jina-reranker-v2-base-multilingual, with rotary or learned positions per checkpoint.
Roadmap
The families candding adds phase by phase, the output modes and quantized variants each model offers, and the runtime work around the catalog.