XLM-RoBERTa (flash)

This family is one checkpoint, jinaai/jina-colbert-v2, and it is the first in the catalog with no dense entry at all: its only output is the multi-vector head the multi-vector page covers, so what follows is the encoder underneath that head, not a pooled-vector model of its own.

What it is

An encoder-only transformer at XLM-RoBERTa-large's own scale: 24 layers, hidden size 1024, 16 attention heads, intermediate size 4096, a 250004-entry SentencePiece Unigram vocabulary and a declared position ceiling of 8194. Every weight is published in bfloat16, and the file size (1 119 027 888 bytes) matches the parameter count (559 497 216) at two bytes each exactly, an independent check that the declared dtype is the real storage rather than a label.

The checkpoint's own readme calls it a ColBERT-style model built on the ALiBi JinaBERT family candding already ships, and every one of its measured tensors contradicts that sentence. The config carries no learned position table and no model_type field at all, so neither claim in that sentence can be checked against the file the way the XLM-RoBERTa page's own repositories can be. What the tensors do carry is XLM-RoBERTa's own tokenizer, XLM-RoBERTa-large's own scale, and a block shape neither BERT nor ALiBi JinaBERT has:

• No position table at all. Positions are rotary and computed at base 10 000, applied to the query and key inside attention rather than added at the embedding layer. roberta.embeddings.token_type_embeddings.weight is a one-row table, the same pitfall the XLM-RoBERTa page already documents for a different repository.
• One fused attention projection. mixer.Wqkv.weight is (3072, 1024) with a bias, query, key and value stacked in that row order across the output dimension, the identical convention the NomicBERT page's own Wqkv pitfall names for a bias-free version of the same shape. Every other family this crate ports reads three separate matrices.
• Post-norm blocks, not pre-norm. Both the attention output and the feed-forward output are added back to their own residual before the layer's own LayerNorm runs — models::bert::BertLayer's own arrangement, with a fused-QKV, rotary-position attention swapped in for BERT's three-matrix one. The reference implementation's own generic block class defaults to pre-norm, and reading that default without reading which value the actual model construction passes would get this backwards; the function this checkpoint's own encoder calls passes the opposite value for every layer.

How candding implements it

models::xlm_roberta_flash::XlmRobertaFlashModel is a fourth family in the registry's own Family enum, XlmRobertaFlash, and the first one that appears only in the multi-vector catalog: Family::from_model_type has no arm for it, because the checkpoint's own config.json has no model_type key for that dispatch to read. Every other family in this catalog is reached by a string a repository's own config publishes; this one is reached by its registry id alone, the same way every catalog entry is looked up, with MultiVectorHead::JinaColbert naming which head loads it as a compile-time constant on that one entry rather than a fact inferred from a fetched file. A future checkpoint with the same gap — no model_type field — cannot silently reach this family by accident: XlmRobertaFlashConfig::ensure_supported refuses any config whose position_embedding_type is not "rotary", and every tensor shape the encoder and its head read is checked against the registry's own declared numbers at load time.

The fused Wqkv matrix is split with layers::attention::split_fused_qkv, the same helper the NomicBERT family already uses for its own fused, bias-free projection. Rotary positions run through layers::rope::RotaryEmbedding, the same module Qwen3 and Gemma3 use for their own causal decoders — this is the first bidirectional consumer of it in the catalog, since nothing indexes a position table to shift or bound here the way a learned one would.

The head is a bias-free Linear(1024, 128) applied to every token position, models::multi_vector::jina_colbert::JinaColbert: the same shape of projection colbert-ir/colbertv2.0 and answerdotai/answerai-colbert-small-v1 already run over a plain BERT encoder, reused here over this family instead of duplicated for it. What is genuinely new to this checkpoint, not shared with either sibling, is that its own self-attention can see the [MASK]-expansion positions a padded query carries past its real content, where both of those checkpoints cannot; the multi-vector page covers this convention, attends_to_mask_tokens, in full.

Pooling and templates

Neither applies here in the dense-pipeline sense: this family has no TextEmbedding entry, no pooling mode and no query or passage string template. What plays their role is the multi-vector head's own query and document markers, "[QueryMarker] " and "[DocumentMarker] ", genuine added vocabulary tokens rather than a reserved WordPiece slot reused the way both BERT-family ColBERT checkpoints do — and the fixed 32-row query padding, the document-side punctuation skiplist, and the late-interaction scoring that consumes the result, all of which the multi-vector page covers rather than this one.

Supported models

The licence is the reason this is the catalog's only entry: it keeps the checkpoint out of .github/workflows/ci.yml and metal.yml, the same rule naver/splade-code-06B's own sparse entry already follows, though both still run locally through just test-models/test-metal. The published dimension variants, jinaai/jina-colbert-v2-96 and jinaai/jina-colbert-v2-64, live in separate repositories and are not in scope. <ModelTable family="xlm_roberta_flash" /> would render empty here on purpose: that component reads the dense catalog alone, and this family has no dense row to show; the live numbers for the one entry it does have are on the multi-vector catalog page and through candding-cli describe jinaai/jina-colbert-v2.

Pitfalls

• The checkpoint's own readme names the wrong family; read the config and the weights, not the prose. position_embedding_type: "rotary" and the absence of position_embeddings.weight anywhere in the header are what rule out both BERT and ALiBi JinaBERT, not the readme's own sentence.
• There is no model_type field to dispatch on, unlike every other family this crate ports; this family is reached only through its registry id and its own MultiVectorHead::JinaColbert constant, never through models::load_encoder's string-keyed table.
• The blocks are post-norm. The reference implementation's own generic block class defaults to pre-norm, and this checkpoint's own construction overrides that default for every layer; taking the class default without reading the override reads the block arrangement backwards.
• Wqkv stacks query, key and value in that row order across the output dimension, with a bias — the identical convention models::nomic_bert's own (bias-free) fused projection already documents, not a fact this checkpoint's tensor names state on their own.
• The mask token this checkpoint's own tokenizer names is <mask>, not the "[MASK]" literal every other multi-vector checkpoint in the catalog happens to use — a fixed-length query tokenizer that assumed the BERT-family spelling filled every padding position with this checkpoint's own <s> embedding instead, a real, measured defect this port's own golden run caught rather than a hypothetical one; crate::tokenizer::read_mask_token resolves the real name from the checkpoint's own tokenizer_config.json instead of assuming it.
• The published tokenizer.json embeds its own truncation at 300 tokens, matching doc_maxlen; a bare Tokenizer::from_file installs that default the same way it installs an embedded padding config, so a reference generator comparing an untruncated tokenizer call against this file's own replay has to clear that default first or the comparison silently disagrees on any input past 300 tokens.
• Every position is rotary and computed from the sequence axis alone; nothing about right padding shifts a real token's own position the way a learned, offset-numbered table would.

References