XLM-RoBERTa (flash)
The XLM-RoBERTa (flash) encoder behind jina-colbert-v2 and jina-reranker-v2-base-multilingual, with rotary or learned positions per checkpoint.
This family is two checkpoints and no dense entry: jinaai/jina-colbert-v2, whose output is the multi-vector head, and jinaai/jina-reranker-v2-base-multilingual, whose output is one rerank score per pair. Both share one encoder and differ only in how they number tokens.
What it is
An encoder-only transformer with XLM-RoBERTa's SentencePiece tokenizer, one fused, biased mixer.Wqkv projection with query, key and value rows in that order, post-norm blocks and a one-row token type table, with every weight published in BF16.
| Checkpoint | Positions |
|---|---|
| jinaai/jina-colbert-v2 | rotary at base 10000, no position table |
| jinaai/jina-reranker-v2-base-multilingual | a learned table of 1026 rows, numbered from pad_token_id + 1, which leaves 1024 positions |
How candding implements it
Neither checkpoint's config.json has a model_type, so Family::XlmRobertaFlash is reached only through a registry entry. Each entry states its position_scheme, since the family's two configuration classes default position_embedding_type differently, and the load runs only when config.json names the same scheme:
The entry's position_scheme and config.json's position_embedding_type | The load |
|---|---|
both rotary | rotary angles on the query and the key: jinaai/jina-colbert-v2 |
both absolute | the learned table, numbered as the XLM-RoBERTa family numbers its own: jinaai/jina-reranker-v2-base-multilingual |
| different, or either absent | refused with CanddingError::PositionSchemeDisagreesWithConfig |
alibi or any other value in config.json | refused with CanddingError::UnsupportedArchitecture |
The ColBERT head is in the multi-vector table. The reranker's head is the XLM-RoBERTa rerankers' RobertaClassificationHead on the <s> position, and its score is the sigmoid of that logit over the best window of a long document.
Pooling and templates
Neither checkpoint pools or carries a template in the dense sense. The ColBERT checkpoint prepends [QueryMarker] and [DocumentMarker] , tokens of its own vocabulary, and the reranker reads the pair template <s> query </s> </s> document </s> with every token type id zero.
Supported models
<ModelTable> reads the dense catalog only, so this family has no generated table; the catalog lists both checkpoints in its multi-vector and rerank tables.
| Model | Layers | Hidden | Weights | License |
|---|---|---|---|---|
| jinaai/jina-colbert-v2 | 24 | 1024 | BF16, 1.12 GB | CC-BY-NC-4.0, non-commercial |
| jinaai/jina-reranker-v2-base-multilingual | 12 | 768 | BF16, 0.56 GB | CC-BY-NC-4.0, non-commercial |
Both licences are non-commercial, so neither checkpoint runs in CI. The dimension variants jinaai/jina-colbert-v2-96 and -64 are separate repositories and are not in the catalog.
Pitfalls
jinaai/jina-colbert-v2's readme calls it an ALiBi JinaBERT model; its config and weights, rotary positions and no position table, say otherwise.- An absent
position_embedding_typehas no safe default; read the reranker's positions the rotary way and every score stays finite while the ranking changes. - The learned table numbered from 0 instead of
pad_token_id + 1moves every score but leaves the model card's ranking unchanged, so a ranking check alone misses it. - The reference's generic block class defaults to pre-norm, and each checkpoint's construction overrides it to post-norm.
- The reranker's
config.jsonnamesXLMRobertaForSequenceClassification, also a stock transformers class, which loads this repository with fresh random tensors and only a warning; testing has the generator's guard. - The query padding token is
<mask>, not the BERT pair's[MASK]; filling it with<s>leaves the vectors plausible and wrong. candding reads the name fromtokenizer_config.json. jinaai/jina-colbert-v2'stokenizer.jsonembeds truncation at 300 tokens; a reference generator has to clear it, or it disagrees on any longer input.