candding
Models

Qdrant/bm25

The lexical BM25 sparse head, hashed rather than learned, with no neural network, no device and no dtype anywhere in it.

Qdrant/bm25 is the one sparse head with no neural network: it splits, filters, stems and hashes the text and weights each stem by term-frequency saturation. It has no checkpoint, vocabulary, device or dtype, and its output is a SparseEmbedding, scored as on the sparse output page.

Using it

candding/examples/bm25.rs
//! `Qdrant/bm25` has no device, no dtype and no checkpoint to fetch, so `build()` cannot fail.

use candding::Bm25TextEmbedding;

fn main() {
  let model = Bm25TextEmbedding::builder().build();
  let passage = model.passage_embed(&["The quick brown fox jumps over the lazy dog."]);
  let query = model.query_embed(&["a quick fox"]);
  println!("indices in the passage row: {}", passage[0].len());
  println!("query . passage = {:.4}", query[0].dot(&passage[0]));
}
ItemBm25TextEmbedding
BuilderBm25TextEmbedding::builder(), with no settings
.build()returns Bm25TextEmbedding, never a Result, since nothing touches a filesystem, a network or a device; the stopword list is compiled into the crate
passage_embed, query_embed, embedtake &[S: AsRef<str>] and return Vec<SparseEmbedding>, with no Result and no batch size; embed is the passage side
.descriptor()a SparseDescriptor with vocab_size 0, default_max_length usize::MAX, weights NoWeights and computation Computed

SparseTextEmbedding::builder("Qdrant/bm25") refuses before any fetch, and its error names Bm25TextEmbedding::builder().

From text to row

StepRule
Cleanevery character that is neither a Unicode word character nor whitespace is removed, and the text is lowercased
Spliton whitespace; a script with no spaces between words, such as Japanese, becomes one token per unbroken run, as in the reference
Filtera token that is bare Unicode punctuation, an English stopword or longer than 40 characters is dropped
Stemthe Snowball English stemmer; a token whose stem is empty is dropped
Indexthe absolute value of the signed 32-bit murmur3 hash of the stem, seed 0, as Python's mmh3.hash computes it
Passage weightcount·(k+1) / (count + k·(1 - b + b·doc_len/avg_len)), with doc_len the number of surviving stems in the text
Query weight1.0 for each unique stem
ConstantValue
k1.2
b0.75
avg_len256

A row is as wide as its distinct stems, anywhere in a 31-bit range, and the hash has no inverse. Two stems that collide on one index keep the larger weight.

Limits

  • Stemming and stopwords are English only, and there is no .language(..) method; the reference takes one of eighteen languages. Splitting and the punctuation filter work on any script.
  • A row's weights are the term-frequency half of BM25 only. The inverse-document-frequency half comes from the vector database's index-time modifier, so query.dot(&passage) between two rows is not the complete BM25 score.
  • A word shared with Qdrant/minicoil-v1 does not land on the same index: miniCOIL hashes only the words its vocabulary lacks, and reduces the same murmur value modulo a range before adding a shift.

How it is checked

There is no checkpoint to compare, so a pinned unit-test suite in the default cargo test --workspace checks it against a hand-reconstructed run of the reference pipeline over the nine fixture inputs, plus cases for the punctuation filter, the length cap and stemming.

On this page