MPNet

MPNet is BERT's encoder with one term added: every attention score picks up a learned per-head bias that depends only on how far the key sits from the query. Everything else about the block — post-LayerNorm residuals, biased projections, the erf GELU, the same feed-forward names — is BERT's, and the bias is what this page is mostly about, because it is the piece that is invisible when you get it wrong.

What it is

An encoder-only transformer over a 30527-entry WordPiece vocabulary, bidirectional attention, learned absolute position embeddings numbered from the padding index upward, no token type table at all, and LayerNorm after each residual addition with an epsilon of 1e-5.

The default length is 384 because the repository's sentence_bert_config.json says max_seq_length: 384 and its card says that longer input is truncated, while the position table holds 512. Raising the builder's max_length past 384 is allowed up to 512 and means you are no longer comparing against what the repository publishes.

The relative position bias

The checkpoint carries one extra tensor that no other catalog model has, encoder.relative_attention_bias.weight, an embedding table of 32 rows by 12 heads. For a query at position i and a key at position j, the offset j - i is mapped to one of the 32 buckets, the bucket's row is read, and the head's value is added to that score.

The bucketing is bidirectional rather than signed: the 32 buckets split into a past half starting at 0 and a future half starting at 16. Inside each half the first eight distances get a bucket each, and everything further is spaced logarithmically, so the model can tell distance 3 from distance 4 exactly and distance 200 from distance 300 not at all. The logarithm is scaled by a maximum distance of 128, and a clamp caps the offset at 15, which in practice means every key 91 or more positions away shares a single bucket per direction.

Bucket 16 is unreachable. The future half starts at 16 and then adds at least 1, while the diagonal lands in bucket 0, so nothing ever selects row 16. The checkpoint confirms it independently: every one of the twelve heads in row 16 is within 0.006 of zero, the only row in the table that is, while the table as a whole ranges from -11.2694 to 3.3707 with a mean absolute value of 1.4479, and the thirty-one reachable rows average 1.4945. That row was initialized and never received a gradient. An implementation with the sign convention backwards puts the diagonal in it and reads an untrained row for every self-attention score, which produces finite, unit-length, entirely wrong vectors.

Where the bias is added, and what that costs

The reference divides the query-key product by the square root of the head size and then adds the bias, at the same point and with the same arithmetic as the padding mask. candding therefore composes the two into one tensor and hands it to the shared attention function, which adds the mask after the scale on both its paths; nothing in the attention code knows this family exists.

Composing them is a selection rather than an addition: a padded key gets exactly the dtype's most negative finite value and a real key gets exactly the bias. Adding them instead would put a bias of -11.27 on top of F16's -65504, which is close enough to the format's edge to be worth not thinking about. The table is loaded in F32 whatever dtype the run is in, the composition happens in F32, and there is one cast at the end, because candle requires the mask tensor to carry the query's dtype.

The cost is shape. A padding mask is the same for every head and every query row, so it is (batch, 1, 1, seq); a bias is not, so the combined tensor is (batch, heads, seq, seq) whenever any row is padded and (1, heads, seq, seq) when none is. On Apple Metal that is not new: a head size of 64 takes candle's fused attention kernel, which wants the mask pre-broadcast to every head anyway. On the matmul path it is new, and the honest number is that at a batch of 32, twelve heads and the 384-token limit an F32 mask is about 226 MB, against roughly 50 KB for a BERT model's narrow one. It doubles an allocation that already exists — the matmul path materializes a scores tensor of exactly that shape and dtype — rather than adding an order of magnitude, and the (1, heads, seq, seq) form is kept whenever nothing in the chunk is padded, which the length-sorted batching produces often.

Position ids and the missing token type table

Position ids start at the padding index plus one and are derived from the row lengths, the same rule the XLM-RoBERTa page explains in full; MPNet uses it with the padding index fixed at 1, written as a literal in the reference rather than read from the config, and candding shares that family's implementation rather than writing a second one.

The embedding is then LayerNorm(word + position) and nothing else. There is no token type table: the config has no type_vocab_size, the reference builds no token_type_embeddings, and the checkpoint has no such tensor. That is the difference from both of candding's other encoder families, and it is the reason this family declares its own config struct instead of reusing BERT's, which would have supplied a defaulted table of two rows to an architecture that has none.

The bias is computed from plain zero-based indices, not from those offset position ids. The reference's bias function accepts position ids and its encoder does not pass any, so it buckets 0..seq. For an unpadded row the two readings agree, because bucketing depends only on differences and the offset cancels; for a padded row they do not, because its ids collapse to the padding index. candding's bias takes a sequence length and a device and nothing else, so there is no id to pass it wrongly.

The tokenizer

WordPiece with a lowercasing BertNormalizer and RoBERTa-style special tokens, which is an unusual pairing and comes from the pretrained checkpoint this model was fine-tuned from. Every input is wrapped as <s> … </s>, so token_count of an empty string is 2; the padding token is <pad>, id 1, which is also the padding id the config declares and the index the position numbering reserves. The vocabulary has both an [UNK] at 104, which the WordPiece model uses, and an unused <unk> at 3.

One near-miss is worth naming because it looks exactly like the catalog's three tokenizer divergences and is not one: tokenizer_config.json says "do_lower_case": true while sentence_bert_config.json says false. sentence-transformers reads the second, and it only controls whether the pipeline lowercases the string before handing it over; it does not rebuild the normalizer. Lowercasing still happens, because the published tokenizer.json carries lowercase: true and candding replays that file. Both implementations lowercase exactly once, in the same place.

Pooling and templates

all-mpnet-base-v2 averages the real tokens and normalizes, and carries no templates, no instruction placeholder and no Matryoshka support; embed, query_embed and passage_embed return the same vectors.

Supported models

Not every model named after MPNet is one

sentence-transformers/paraphrase-multilingual-mpnet-base-v2 declares "model_type": "xlm-roberta" and "architectures": ["XLMRobertaModel"]. Its name is not its architecture, and the repository's own files do not say where the name comes from; what the checkpoint says is unambiguous: it has a token type table, absolute positions, no relative attention bias at all, and a weight file the size of intfloat/multilingual-e5-base's. It runs on the XLM-RoBERTa family, and the catalog lists it there. A repository's model_type decides its family, never its name.

Pitfalls

The three cosines below were measured against the reference implementation while porting the family, each by making the mistake on purpose and running the real checkpoint; two of them are pinned by unit tests, and the first is not, because reproducing it needs the weights.

• The relative position bias is added after the query-key product is scaled, not before. Adding it before multiplies every bias value by the head size, sharpens the attention distribution and errors nowhere; against the reference it measures a minimum cosine of 0.7056.
• The offset is key - query. Reading it the other way, or transposing the bias, swaps the past and future halves and moves the diagonal into the untrained bucket 16: minimum cosine 0.5371.
• Dropping the bias entirely leaves finite, unit-length vectors and a minimum cosine of 0.6449.
• The bucket count in the config sizes the table; the bucketing itself uses the reference's own default arguments of 32 buckets and a maximum distance of 128. The two agree only because this checkpoint declares 32, so candding refuses any other value at load rather than silently disagreeing with the reference.
• The bias is the same tensor for every layer and every row of the batch and depends only on the sequence length; computing it per layer is waste, and computing it from the padded position ids is wrong for any padded row.
• There is no token type table. A config struct borrowed from BERT would default one into existence.
• The epsilon is 1e-5, not BERT's 1e-12, and the reference config class defaults it to 1e-12; read it from the checkpoint.
• pooler.dense.* and the embeddings.position_ids buffer are in the file and the sentence-transformers pipeline never runs either; the family never requests them.
• The default length is 384 while the position table holds 512, and the two are different numbers for different reasons: the first is the repository's published max_seq_length and the second is what the weights can index.

References