XLM-RoBERTa

XLM-RoBERTa is BERT's encoder with one rule changed: positions are numbered from the padding id upward instead of from zero. The multilingual-e5 base and large models, BAAI/bge-m3 and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 run on it, as do two rerankers and bge-m3's sparse and multi-vector heads.

What it is

BERT's encoder, with its attention block, erf GELU, biases and weight names, over a 250002-entry SentencePiece Unigram vocabulary, with a single-row token type table and an epsilon of 1e-5. The tokenizer wraps every input in <s> and </s>, so token_count of an empty string is 2, and pads with <pad>, id 1.

How candding implements it

Position ids come from the token ids, as in the reference: the running count of non-padding ids is added to the padding id, so a row's first real token is position 2 and padding stays at 1. A real <pad> in the text is numbered as padding too, and every later token one lower than its column.

BAAI/bge-m3 publishes only a pytorch_model.bin, read tensor by tensor and slower to load than memory-mapped safetensors. Its default length costs gigabytes per layer on the matmul path, so on the CPU embed long documents one at a time; devices and dtypes has the sizes.

A reranker pair is <s> query </s></s> document </s> with every token type id zero. BAAI/bge-reranker-v2-m3 has bge-m3's shape and defaults to FlagEmbedding's 512 tokens, although its tokenizer declares 8192.

Pooling and templates

The e5 model card puts query: on both sides for tasks other than retrieval, so a symmetric comparison uses query_embed twice. None of these models has an instruction placeholder or Matryoshka support, and paraphrase-multilingual-mpnet-base-v2 ships no Normalize module.

Supported models

Pitfalls

• Position ids start at the padding id plus one and come from the token ids; numbering from zero shifts every hidden state and still yields unit-length vectors.
• The single token type row is not zero, so the lookup runs on every forward.
• The epsilon is 1e-5, not BERT's 1e-12; read it from the config.
• A repository's model_type decides its family, never its name: intfloat/multilingual-e5-small declares bert and runs on BERT, and paraphrase-multilingual-mpnet-base-v2 declares xlm-roberta and runs here, not on MPNet.
• paraphrase-multilingual-mpnet-base-v2 defaults to its repository's 128 tokens and truncates longer text silently, as the reference does.
• candding and the golden references replay the published tokenizer.json, which sentence-transformers and FlagEmbedding rebuild with a pre-tokenizer step: for intfloat/multilingual-e5-small, every BAAI/bge-m3 head and BAAI/bge-reranker-base, the two differ by one trailing token on an input whose normalized form ends in whitespace, and nowhere else.
• A live FlagReranker call also builds a different input for an empty document; the rerank page describes that case.
• pooler.dense.* is in every checkpoint and never runs.

References