Multi-vector output
What a multi-vector embedding is, the conventions its four checkpoints disagree on, late-interaction scoring, and each checkpoint's status per dtype.
A multi-vector embedding is a matrix with one row per surviving token instead of one pooled vector. A query and a document are compared by late interaction, MaxSim: for each query row, the largest dot product against any document row, summed over the query rows.
let model = candding::MultiVectorTextEmbedding::builder("colbert-ir/colbertv2.0").build()?;
let query = model.query_embed(&["a quick fox"], None)?;
let passage = model.passage_embed(&["The quick brown fox jumps over the lazy dog."], None)?;
let descriptor = model.descriptor();
let score = candding::multi_vector::late_interaction_score_for(
descriptor.score_reduction,
&query[0],
&passage[0],
descriptor.masked_document_row_fill,
);query_embed and passage_embed return one MultiVectorEmbedding per text, since a checkpoint can tokenize and pad a query differently from a document; embed is the document side. Every row is L2-normalized, unless the builder's normalize(false) or the CLI's --no-normalize asks for the rows as the projection writes them. The entries live in their own catalog, registry::multi_vector.
The four checkpoints
| Checkpoint | Encoder | Head | Projection | Bias | Keeps the first row | Score reduction | Query rows | Query attends to its mask tokens |
|---|---|---|---|---|---|---|---|---|
| BAAI/bge-m3 | XLM-RoBERTa | M3Colbert | 1024 to 1024 | yes | no | MeanByQueryLength | its own length | no |
| colbert-ir/colbertv2.0 | BERT | StanfordColbert | 768 to 128 | no | yes | Sum | 32 | no |
| answerdotai/answerai-colbert-small-v1 | BERT | StanfordColbert | 384 to 96 | no | yes | Sum | 32 | no |
| jinaai/jina-colbert-v2 | XLM-RoBERTa (flash) | JinaColbert | 1024 to 128 | no | yes | Sum | 32 | yes |
Each convention is descriptor data, keeps_cls_row, score_reduction and attends_to_mask_tokens, not a rule in the head or the scoring function. BAAI/bge-m3's reference divides the MaxSim sum by the query's real row count, so the wrong reduction scales a score by that count.
vidore/colpali-v1.3 is on the roadmap, with the image models.
Query padding, markers and punctuation
The three ColBERT-style checkpoints pad every query to exactly 32 rows with the mask token, [MASK] for the BERT pair and <mask> for jinaai/jina-colbert-v2, read from tokenizer_config.json; the 32 is the checkpoint's contract, not a max_length. Each document keeps its own row count up to the truncation cap.
| Checkpoints | Query marker | Document marker |
|---|---|---|
| colbert-ir/colbertv2.0, answerdotai/answerai-colbert-small-v1 | [unused0] | [unused1] |
| jinaai/jina-colbert-v2 | [QueryMarker] | [DocumentMarker] |
All three drop punctuation rows from documents, so the surviving rows are not a prefix of the sequence. BAAI/bge-m3 has no fixed query length, no markers and no punctuation skiplist.
Late-interaction scoring
multi_vector::late_interaction_score takes an optional mask per side, None meaning every row is real, as every catalog checkpoint returns; late_interaction_score_for applies a checkpoint's own ScoreReduction and masked_document_row_fill.
| Edge case | Score |
|---|---|
| a document with no real rows | -1e9, so it sorts last; never rescaled by the reduction |
| a query with no real rows against a real document | 0.0 |
| a masked document row | filled with the descriptor's MaskedRowFill before the maximum: the dtype minimum for the ColBERT-style three, zero for BAAI/bge-m3 |
| a masked query row | left out of the sum |
No shape panics. BAAI/bge-m3's multi-vector head tokenizes with the published tokenizer.json; the XLM-RoBERTa page has the trailing-whitespace case where that differs from FlagEmbedding.
Verification
Each checkpoint's golden suite scores all 81 query-document pairs of the nine fixture inputs against its reference. F32 is verified on the CPU and on Metal for all four, and on CUDA for all but jinaai/jina-colbert-v2, which no workflow fetches; the catalog shows each dtype's CUDA cell.
| Checkpoint | F16 | BF16 |
|---|---|---|
| BAAI/bge-m3 | unsupported: cosine 0.9964 on the CPU and 0.9959 on Metal, against a 0.998 floor | unsupported: cosine 0.9939 on Metal, against a 0.995 floor |
| colbert-ir/colbertv2.0, answerdotai/answerai-colbert-small-v1, jinaai/jina-colbert-v2 | verified on the CPU and on Metal | verified on Metal |
BF16 is refused on the CPU for all four, as for every model. BAAI/bge-m3's sparse head passes F16 on Metal on the same encoder; the multi-vector head's miss has no established cause.
BAAI/bge-m3 has a dense, a sparse and a multi-vector entry, one set of weights behind three rows with their own statuses. The other three checkpoints are multi-vector entries only, so the catalog's multi-vector table lists them and the family pages' model tables do not.
Sparse output
What a sparse embedding is, what each of the six sparse heads indexes and computes, how two rows are scored, and which dtypes each head supports.
Rerank output
What a reranker returns for a query and its documents, what each checkpoint's score means, how long inputs are cut, and each checkpoint's status.