Qdrant/bm25

Qdrant/bm25 is the only one of candding's six sparse heads with no neural network anywhere in its own computation: every step is string or scalar arithmetic over the input text itself, never a forward pass through a checkpoint, which is what makes it lexical rather than learned in the sense every other head in the catalog is. A reader arriving here from the catalog meets a row with no dimension and no pooling because there is nothing to pool: there is no vocabulary, no tokenizer in the encoder sense, no device and no numeric format anywhere in what this model computes.

Using it

//! `Qdrant/bm25` has no device, no dtype and no checkpoint to fetch, so `build()` cannot fail.

use candding::Bm25TextEmbedding;

fn main() {
  let model = Bm25TextEmbedding::builder().build();
  let passage = model.embed(&["The quick brown fox jumps over the lazy dog."]);
  let query = model.embed_query(&["a quick fox"]);
  println!("indices in the passage row: {}", passage[0].len());
  println!("query . passage = {:.4}", query[0].dot(&passage[0]));
}

Its own entry point is a second, smaller type beside the one every other sparse model shares: Bm25TextEmbedding and its builder Bm25TextEmbeddingBuilder, both exported at the crate root next to SparseTextEmbedding/SparseTextEmbeddingBuilder rather than folded into them. The builder carries none of the methods a device or a numeric format would need — no .device(..), no .dtype(..), no .max_length(..), no .batch_size(..), no .revision(..) — because none of those concepts apply to this model, not because a default sits behind an omitted call; .build() itself takes no argument and returns Bm25TextEmbedding directly, never a Result, since nothing between the call and a usable value touches a filesystem, a network or a device — the one stopword list this model needs is compiled into the binary. embed and embed_query follow the same shape: both take &[S: AsRef<str>] and return Vec<SparseEmbedding> directly, with no Result and no batch_size parameter, since every text is independent work rather than a chunk sharing a device tensor with the rest of a batch. .descriptor() returns the same SparseDescriptor every sparse entry exposes, and on this one every field that would otherwise describe a checkpoint says instead that there is none: vocab_size reads 0, default_max_length reads usize::MAX, weights reads NoWeights, and computation reads Computed rather than Encoded. Reaching for it the way every other model is reached fails on purpose: SparseTextEmbedding::builder("Qdrant/bm25") refuses before attempting any fetch, and the error names Bm25TextEmbedding::builder() as the fix — what Computed means beside Encoded, and the rest of that refusal's own mechanics, are covered on the sparse output page.

What the index means

A Qdrant/bm25 row is a SparseEmbedding like any other head's — see sparse output for the type itself and how two rows are scored against each other — but what its own index names is not what any other head's index names. Every other sparse head in the catalog indexes a position: a WordPiece id, a SentencePiece id, a byte-level BPE id, or, for one head, a private word id. Qdrant/bm25 indexes a hash instead: the absolute value of a signed 32-bit murmur hash of an already-stemmed word, used as the index unmodified, landing anywhere across a 31-bit range with no vocabulary bounding it on either side.

That last row is not hypothetical: models::sparse::bm25's own pipeline never special-cases a collision, and when two stems do land on the same index, SparseEmbedding::from_pairs keeps the larger of the two weights — the same fold rule every other head in the catalog already applies to its own duplicate indices, not a rule written for this model alone.

Language

Splitting the text into candidate tokens is not English-specific: remove_non_alphanumeric and tokenize both test Unicode general categories that hold for any script, checked in this port's own tests against Hungarian, Japanese and accented Latin text alongside English, not only assumed from the reference's own regexes. One limitation survives from the reference's own regex-based tokenizer rather than being introduced by this port: a script with no whitespace between words — Japanese is the tested case — tokenizes as one undivided run per unbroken block of word characters, not one token per linguistic word. The punctuation filter that runs next, dropping a token that is exactly one Unicode punctuation character, is the same general-category check, and is not English-specific either. Stemming and stopword removal are the two steps that stay English regardless of the input: stem is hardcoded to the Snowball English algorithm and runs on every surviving token unconditionally, and the one stopword list compiled into candding is the reference's own English file, with no .language(..) builder method to ask for a different one. The published reference is not English-only by policy the way this port is: its own repository ships stopword files for thirty languages, though its own Python constructor only ever names eighteen of them, its own hardcoded list; the other twelve are real files in that same repository the reference's own API cannot reach at all, and this port exposes none of the eighteen today.

Scoring

A document's surviving stems come from the same four-step filter every time: drop a token that is bare Unicode punctuation, drop one that is an English stopword, drop one longer than 40 characters, then stem what is left and drop it again if stemming produced an empty string. Each surviving stem is then counted and weighted by term-frequency saturation, count·(k+1) / (count + k·(1 - b + b·doc_len/avg_len)), where doc_len is the number of surviving stemmed tokens in that document — not the raw pre-filter count, and not the count of distinct stems — at this model's own constants.

k and b happen to match Qdrant/minicoil-v1's own constants and the formula shape is the same declared shape, but avg_len does not match, and both models declare all three independently rather than one importing the other's, so a later change to either is not meant to silently move the other. A query never runs this formula at all: every unique surviving stem is hashed at a flat weight of 1.0, the same network-free shortcut Bm42 takes for its own query path, unlike Qdrant/minicoil-v1, which still runs its real encoder to weight a query — uniformly, but by a real forward pass, not a shortcut. Bm42's own passage score is not this formula at all: it sums a pooled attention weight per stem, keeps the maximum among duplicates, and rescales with ln(1 + value)^0.5, a value read out of a real encoder's last layer rather than counted from the text — the flat-1.0 query shortcut is the one thing the two heads' scoring behavior has in common. The weight this model produces is only the term-frequency half of the published BM25 formula: the reference's own catalog record for this model declares that the inverse-document-frequency half is supplied externally, by a vector database's own index-time modifier, not computed here, so a raw .dot() between two Bm25TextEmbedding outputs is a real number but not the complete BM25 score its name might suggest.

Minicoil's fallback is a different hash

Qdrant/minicoil-v1 is the only other head in the catalog with a hash anywhere in its own index, and its own published README names Qdrant/bm25's score as what an unrecognized word falls back to; candding's own registry entry for minicoil records this as wrong, checked against both references' own source rather than against the README's own prose. The two hashes start from the same value — abs(murmur3_x86_32(stem, seed=0)), computed the way Python's signed mmh3.hash computes it, over the identical Snowball-English stem — and diverge from there. Qdrant/minicoil-v1's own hashed_index reduces that value modulo a constant and adds a fixed shift, 128000 against its own published word matrix, before using the result as an index; Qdrant/bm25's own hash does neither step and is the value used as the index directly. The two are written and tested as two independent functions in the crate for exactly this reason: collapsing them back into one, the way the README's own claim implies they already are, would let a future change to either one silently change the other too. Hashing plays a different role in the two heads as well, not only a different formula: it is Qdrant/minicoil-v1's own fallback, reached only for a word its private, shipped vocabulary fails to recognize, with every recognized word getting a dense index instead; it is Qdrant/bm25's entire index, since this model has no vocabulary of any kind to check a word against first. A caller who assumes a shared word lands on one shared index across the two heads — the assumption the README's own claim invites — would be wrong twice over: the number differs by construction, and for most of Qdrant/minicoil-v1's own vocabulary, hashing never runs at all.

How it's checked

Nothing about this head runs through the golden-vector harness the other five sparse heads use, and nothing should: that harness fetches a checkpoint and builds a tokenized input, and this head does neither. Its own proof is a pinned unit-test suite inside models::sparse::bm25, checked once against a hand-reconstructed run of the reference's own pipeline over the catalog's shared fixtures plus targeted cases for its own punctuation filter, length cap and stemming, and it runs unconditionally in cargo test --workspace rather than behind the feature flag every fetched checkpoint's own golden run needs.