candding
Models

XLM-RoBERTa (flash)

The XLM-RoBERTa (flash) encoder behind jina-colbert-v2 and jina-reranker-v2-base-multilingual, with rotary or learned positions per checkpoint.

This family is two checkpoints and no dense entry: jinaai/jina-colbert-v2, whose output is the multi-vector head, and jinaai/jina-reranker-v2-base-multilingual, whose output is one rerank score per pair. Both share one encoder and differ only in how they number tokens.

What it is

An encoder-only transformer with XLM-RoBERTa's SentencePiece tokenizer, one fused, biased mixer.Wqkv projection with query, key and value rows in that order, post-norm blocks and a one-row token type table, with every weight published in BF16.

CheckpointPositions
jinaai/jina-colbert-v2rotary at base 10000, no position table
jinaai/jina-reranker-v2-base-multilinguala learned table of 1026 rows, numbered from pad_token_id + 1, which leaves 1024 positions

How candding implements it

Neither checkpoint's config.json has a model_type, so Family::XlmRobertaFlash is reached only through a registry entry. Each entry states its position_scheme, since the family's two configuration classes default position_embedding_type differently, and the load runs only when config.json names the same scheme:

The entry's position_scheme and config.json's position_embedding_typeThe load
both rotaryrotary angles on the query and the key: jinaai/jina-colbert-v2
both absolutethe learned table, numbered as the XLM-RoBERTa family numbers its own: jinaai/jina-reranker-v2-base-multilingual
different, or either absentrefused with CanddingError::PositionSchemeDisagreesWithConfig
alibi or any other value in config.jsonrefused with CanddingError::UnsupportedArchitecture

The ColBERT head is in the multi-vector table. The reranker's head is the XLM-RoBERTa rerankers' RobertaClassificationHead on the <s> position, and its score is the sigmoid of that logit over the best window of a long document.

Pooling and templates

Neither checkpoint pools or carries a template in the dense sense. The ColBERT checkpoint prepends [QueryMarker] and [DocumentMarker] , tokens of its own vocabulary, and the reranker reads the pair template <s> query </s> </s> document </s> with every token type id zero.

Supported models

<ModelTable> reads the dense catalog only, so this family has no generated table; the catalog lists both checkpoints in its multi-vector and rerank tables.

ModelLayersHiddenWeightsLicense
jinaai/jina-colbert-v2241024BF16, 1.12 GBCC-BY-NC-4.0, non-commercial
jinaai/jina-reranker-v2-base-multilingual12768BF16, 0.56 GBCC-BY-NC-4.0, non-commercial

Both licences are non-commercial, so neither checkpoint runs in CI. The dimension variants jinaai/jina-colbert-v2-96 and -64 are separate repositories and are not in the catalog.

Pitfalls

  • jinaai/jina-colbert-v2's readme calls it an ALiBi JinaBERT model; its config and weights, rotary positions and no position table, say otherwise.
  • An absent position_embedding_type has no safe default; read the reranker's positions the rotary way and every score stays finite while the ranking changes.
  • The learned table numbered from 0 instead of pad_token_id + 1 moves every score but leaves the model card's ranking unchanged, so a ranking check alone misses it.
  • The reference's generic block class defaults to pre-norm, and each checkpoint's construction overrides it to post-norm.
  • The reranker's config.json names XLMRobertaForSequenceClassification, also a stock transformers class, which loads this repository with fresh random tensors and only a warning; testing has the generator's guard.
  • The query padding token is <mask>, not the BERT pair's [MASK]; filling it with <s> leaves the vectors plausible and wrong. candding reads the name from tokenizer_config.json.
  • jinaai/jina-colbert-v2's tokenizer.json embeds truncation at 300 tokens; a reference generator has to clear it, or it disagrees on any longer input.

References

Conneau et al. (2019). Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116
Khattab and Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv:2004.12832

On this page