XLM-RoBERTa

XLM-RoBERTa is BERT's encoder with one rule changed: positions are numbered from the padding id upward instead of from zero, and everything that follows from that. The catalog's multilingual-e5 models run on it, and the smallest of them does not, which is the first thing this page explains.

What it is

An encoder-only transformer with a 250002-entry SentencePiece Unigram vocabulary, bidirectional attention, learned absolute position embeddings, a single-row token type table and LayerNorm after each residual addition. The attention block, the erf GELU, the biases and the weight names are BERT's; the epsilon is 1e-5 rather than BERT's 1e-12, and the position numbering is the difference that decides whether the vectors match.

The tokenizer wraps every input in <s> and </s>, so token_count of an empty string is 2, and it pads with <pad>, id 1, which is also the padding id config.json declares.

Position ids are the whole difference

nn.Embedding(max_position_embeddings, hidden, padding_idx=1) reserves the rows up to and including the padding id, and the reference builds the ids as the cumulative sum of the attention mask, multiplied by that mask, plus the padding id. Under right padding that has a closed form: the first real token of a row is at index 2, the last real token of an n-token row at n + 1, and every padded column stays at 1.

Three consequences follow.

The stored number of positions is the sequence limit plus two, which is why the repositories publish 514 positions for a max_seq_length of 512 (the e5 models) or 8194 for 8192 (bge-m3), and why Encoder::max_position_embeddings returns the limit rather than the stored count. Returning the stored count would let a caller ask for a length whose last position id indexes past the table.

The row at the padding index is the position embedding of every padded column. It is zero in the checkpoint because nn.Embedding zero-initializes it and never gives it a gradient, and candding does not rely on that: padded columns are floored out of attention as keys and pooling reads only real positions, so all that matters is that the index is in range.

Broadcasting one row of consecutive ids over the whole batch would give the same pooled vectors, because real tokens get the same ids either way and padded columns are discarded. candding builds the ids per row anyway, so that the hidden states of the padded rows match the reference too, which is what a layer dump compares when a golden test misses.

Not every multilingual-e5 model is one of these

intfloat/multilingual-e5-small declares "model_type": "bert" and "architectures": ["BertModel"]. It is a multilingual MiniLM distilled from XLM-R that carries the XLM-R tokenizer, with 512 absolute positions numbered from zero, a two-row token type table and an epsilon of 1e-12. It runs on the BERT family with no code of its own, and the catalog lists it there. The family a repository gets comes from its model_type, never from its name: numbering that checkpoint from index 2 would read the wrong rows for every input and run off the end of the table for a full-length one.

Pooling and templates

Both models average the real tokens and normalize, and both carry the prefixes their model card states.

The card gives query: as the right prefix for tasks other than retrieval as well, so a symmetric similarity comparison uses query_embed on both sides. There is no instruction placeholder and no Matryoshka support.

bge-m3

The third model on this family shares the body and differs in three places. It takes the [CLS] state instead of averaging, it carries no templates at all because its card states that the model no longer requires instructions on queries, and it stores 8194 positions for a sequence limit of 8192.

That length is the repository's own max_seq_length and the value the reference generator uses, and it is expensive on the matmul path: a single input at the limit with sixteen heads materializes a scores tensor of about four gigabytes per layer. Apple Metal never sees it, because a head size of 64 takes candle's fused attention kernel, which never materializes the scores; the same kernel wants the padding mask pre-broadcast to every head, which is another sixteen planes when a batch has rows of different lengths and nothing at all when it does not. On CPU, long inputs are best embedded one at a time.

Weights in a torch pickle

bge-m3 is the one catalog repository that publishes no safetensors: its encoder is a pytorch_model.bin, beside an ONNX export. The registry entry records that, and just fetch BAAI/bge-m3 adds the root pickle to the include patterns for that model alone; the other catalog repositories that also ship a pytorch_model.bin beside their safetensors keep downloading only the safetensors. The loader falls back to the pickle when neither a model.safetensors nor a shard index is present, and the weights go through the same cast-on-CPU path the safetensors take, because candle's own pickle backend casts on the device instead.

The reader reopens the archive once per tensor and the checkpoint holds 391 of them, so the first load is slower than a memory-mapped safetensors load of the same size; on the machine that verified this entry it took about 1.8 seconds on CPU.

Two heads sit beside the encoder as separate files, sparse_linear.pt and colbert_linear.pt, and candding does not load either: the sparse head is a per-token linear whose relu'd output is scattered into the vocabulary by input token id, and the multi-vector head is a linear over the token states with the [CLS] position dropped. The roadmap lists both under the phases that add those output modes.

Supported models

Pitfalls

• Position ids start at the padding id plus one and come from the attention mask; numbering from zero shifts every hidden state while leaving the vectors unit-length and plausible.
• The stored position count is the sequence limit plus the same offset; the accessor the builder caps max_length with must return the limit.
• type_vocab_size is 1 and the single row is not zero: it has an L2 norm of about 0.52 for the base model and 0.59 for the large one, and a constant added before a LayerNorm moves the normalized output. The lookup runs on every forward.
• The epsilon is 1e-5, not BERT's 1e-12; read it from the config.
• The two sizes share one tokenizer.json byte for byte, and it differs from the one intfloat/multilingual-e5-small ships: the shared file has a WhitespaceSplit pre-tokenizer that drops a whitespace-only input, where the small model's normalizer collapses runs of spaces and its Metaspace pre-tokenizer keeps one. References are generated per model id for that reason.
• pooler.dense.* is in every checkpoint and the sentence-transformers pipeline never runs it; the family never requests it, and neither the pooler nor the embeddings.position_ids buffer costs anything at load.
• bge-m3 pools the [CLS] state where its multilingual-e5 siblings average; the pooling comes from 1_Pooling/config.json and nothing about the family implies one or the other.
• bge-m3's default length is 8192, so a long input on the matmul path costs gigabytes per layer; embed long documents one at a time on CPU.
• bge-m3's pytorch_model.bin is the only weight file it publishes; a fetch that pulls only safetensors leaves the snapshot with configs and no weights, and the failure surfaces at load rather than at download.

References