Multi-vector output

A multi-vector embedding is a matrix, one row per surviving token instead of one pooled row per input, and the late-interaction score that consumes it runs a query's rows against a document's asymmetrically: every document is reduced by the maximum of its own rows before every query is reduced by the sum of its own.

pub struct MultiVectorEmbedding { /* one flat, row-major buffer plus the row width */ }

impl MultiVectorEmbedding {
  pub fn from_rows(dim: usize, rows: impl IntoIterator<Item = Vec<f32>>) -> Self;
  pub fn dim(&self) -> usize;
  pub fn row_count(&self) -> usize;
  pub fn is_empty(&self) -> bool;
  pub fn rows(&self) -> impl Iterator<Item = &[f32]>;
}

MultiVectorTextEmbedding::builder(id).build()?.embed(texts, None)? returns Vec<MultiVectorEmbedding>, one per-token matrix per input text, the multi-vector counterpart of TextEmbedding::embed's Vec<Vec<f32>> and SparseTextEmbedding::embed's Vec<SparseEmbedding>; embed_query is the query-side counterpart, since a checkpoint's own contract can tokenize and pad a query differently from a document, not only score it differently once both are embedded.

Three checkpoints ship this output today: BAAI/bge-m3's own third head beside its dense and sparse ones, and colbert-ir/colbertv2.0 and answerdotai/answerai-colbert-small-v1, both plain BERT encoders candding already runs. jinaai/jina-colbert-v2 and vidore/colpali-v1.3 are on the roadmap instead, for reasons that have nothing to do with this output kind itself.

The two conventions

The three checkpoints disagree about two things, in ways that are checkpoint data rather than a rule this crate could bake into either the head code or the scoring function.

BAAI/bge-m3's head drops the classification row before it ever projects a hidden state, so an input that tokenizes to nothing but its two boundary tokens still returns exactly one row, never zero; both Stanford-ColBERT-style checkpoints keep that row instead, so the same shape of input returns three. Neither convention is a default the other overrides: registry::multi_vector::MultiVectorEntry::keeps_cls_row states the fact per checkpoint, and each head's own row-selection code reads only its own entry's value.

The score reduction disagrees the same way. BAAI/bge-m3's own reference divides the raw late-interaction sum by the query's real, unpadded row count by default; the libraries serving both Stanford-ColBERT-style checkpoints default to the plain, unscaled sum instead. multi_vector::ScoreReduction names the two (Sum, MeanByQueryLength) and multi_vector::late_interaction_score_for reads a checkpoint's own value from its descriptor rather than assuming either — a mutation that applies BAAI/bge-m3's own division to a Stanford-ColBERT-style checkpoint's sum instead of its own reproduces the opposite checkpoint's convention exactly: a real, measured score divides down by a factor of 32, the fixed row count every query on that head carries regardless of how many real words the query's own text has.

Late-interaction scoring

let model = candding::MultiVectorTextEmbedding::builder("colbert-ir/colbertv2.0").build()?;
let query = model.embed_query(&["a quick fox"], None)?;
let passage = model.embed(&["The quick brown fox jumps over the lazy dog."], None)?;
let score = candding::multi_vector::late_interaction_score_for(
  model.descriptor().score_reduction,
  &query[0], None,
  &passage[0], None,
);

late_interaction_score is the reduction itself, MaxSim: for every real row of the query, the largest dot product against any real row of the document, summed over query rows — asymmetric by construction, since the document's rows are reduced by maximum before the query's are reduced by sum, not a symmetric approximation of one shared formula. A document with no real rows scores f32::MIN regardless of the query, so it always sorts last; a query with no real rows against a real document scores 0.0, a sum with nothing in it; neither shape panics. late_interaction_score_for wraps the same reduction with a checkpoint's own ScoreReduction, without ever rescaling the empty-document sentinel — dividing a fixed, meant-to-sort-last value by a query's own row count would still let it drift as that count grew, exactly what the sentinel exists to avoid.

A mask on either side marks which rows are real; None means every row is real, the shape every checkpoint in the catalog actually produces today, since a punctuation skiplist removes a row by a masked gather before a MultiVectorEmbedding is ever built rather than handing the scoring function a row to ignore. A document-side padding row is excluded from ever winning a maximum by filling it with the dtype minimum before the reduction; a query-side padding row is excluded from the sum entirely, the asymmetric treatment the two sides already get carried one step further.

Query padding and the punctuation skiplist

colbert-ir/colbertv2.0 and answerdotai/answerai-colbert-small-v1 share machinery beyond the projection shape, all of it measured rather than assumed from the model card. Every query is padded to exactly 32 rows, filled past its own real content with the mask token rather than the ordinary padding token — a fixed contract of the checkpoint, not a caller-adjustable max_length, which is why MultiVectorTextEmbedding's builder keeps a second, dedicated query tokenizer beside its ordinary document one. Documents are never padded to a common width; each one keeps its own real row count up to the checkpoint's own truncation cap. Both checkpoints prepend a role marker before tokenization, "[unused0] " for a query and "[unused1] " for a document, registered as a special token so it tokenizes as one piece rather than shattering across ordinary punctuation splitting. Both drop punctuation from the document side by a masked gather over the surviving positions, not a truncation: the surviving rows are not a prefix of the original sequence, so a port that truncated instead would agree on a short input, where nothing follows the last real token, and diverge on a real one, where it does not.

BAAI/bge-m3 carries none of this: its own pipeline has no fixed-length query padding, no role marker and no punctuation skiplist, which is why query_marker, document_marker and query_length all read None on its own registry entry.

The one required divergence

One of the nine shared fixtures, whitespace, tokenizes to three ids through the published tokenizer.json candding replays and two through the plain tokenizer the reference uses — the same disagreement BAAI/bge-m3's sparse head already carries on its own whitespace row, for the same tokenizer reason. Every query-document pair touching that one fixture disagrees on row count by construction, not by a numeric miss a wider tolerance could close, so BAAI/bge-m3's own golden suite requires the disagreement rather than excluding it: of the 81 query-document pairs it checks, 64 are compared by ordinary numeric score agreement, and the other 17 — every pair touching the diverging fixture, as a query, as a document, or both — assert the exact recorded row count on each side instead. Neither Stanford-ColBERT-style checkpoint needs this: their own published tokenizer.json agrees with the reference on every one of the nine fixtures, so both of their golden suites check all 81 pairs by ordinary numeric agreement.

Verification status

Full precision is verified on both the processor and the accelerator for all three checkpoints. Half precision splits by checkpoint, and every status below is measured, not assumed from a sibling entry — some of the cells pass by asserting a refusal rather than by comparing a number, and the two are marked separately rather than folded into one count.

BAAI/bge-m3's own F16 misses on both devices, so the accelerator does not rescue it the way it does for that same checkpoint's sparse head, whose own F16 is accelerator-only and verified there: the two heads share an encoder but not a half-precision outcome. Both Stanford-ColBERT-style checkpoints clear every dtype on both devices instead, a cleaner result than BAAI/bge-m3's own head, plausibly because every row on both heads is L2-normalized before any half-precision comparison runs and both encoders are markedly smaller than bge-m3's — a real difference between the checkpoints, not a gap in how thoroughly either was measured.

A third catalog, not a mode

Multi-vector output is a third, independent object graph beside the dense and sparse ones, the same shape the sparse page describes for its own catalog: MultiVectorModel is its own trait, separate from Encoder and from SparseModel, because a multi-vector head's forward pass already is the whole postprocessing step, projection and normalization included, with no pooling stage outside it to share.

The three catalogs are independent lists an id can sit in any combination of: BAAI/bge-m3 is a dense entry in registry, a second entry in registry::sparse, and a third in registry::multi_vector, one set of encoder weights behind three independently statused catalog rows. colbert-ir/colbertv2.0 and answerdotai/answerai-colbert-small-v1 sit in registry::multi_vector alone — their own BERT-family encoders carry no dense catalog row of their own — which is why the BERT family page covers their multi-vector head in prose rather than in its own <ModelTable>, the same way it already covers the family's two sparse heads. Registering a multi-vector entry never touches the dense catalog: its own count stays 35 models across nine families, exactly as before this phase.

website/public/models.json carries a third multi_vector array beside models and sparse, and the catalog page renders it as a third table beneath the other two, with its own columns — a multi-vector row has no dim, pooling, normalize or vocabulary to put in either existing table. The CLI page has the exact fields list-models and describe print for a multi-vector entry, and how a fetch phase:4 reaches all three registries at once.