MPNet

MPNet is BERT's encoder with one term added: every attention score picks up a learned per-head bias that depends only on how far the key sits from the query. The catalog's one MPNet model is sentence-transformers/all-mpnet-base-v2.

What it is

An encoder-only transformer with BERT's block: post-LayerNorm residuals, biased projections, the erf GELU and the same feed-forward names. It differs from BERT in four places:

The tokenizer is a lowercasing WordPiece with RoBERTa-style special tokens, <s> … </s>, so token_count of an empty string is 2, and <pad> is id 1.

How candding implements it

The bias table, encoder.relative_attention_bias.weight, holds 32 buckets per head: the offset from query to key falls in a past or a future half, exact for the first eight distances and logarithmic up to 128. candding adds the bias and the padding mask as one F32 tensor, cast once to the query's dtype and shared by every layer and row; devices and dtypes has its memory cost.

Pooling and templates

all-mpnet-base-v2 averages the real tokens and has no templates, no instruction placeholder and no Matryoshka support, so embed, query_embed and passage_embed return the same vectors. Its default length is the repository's max_seq_length of 384, below the position table's 512.

Supported models

sentence-transformers/paraphrase-multilingual-mpnet-base-v2 declares "model_type": "xlm-roberta" and has no relative bias, so it runs on the XLM-RoBERTa family and is listed there.

Pitfalls

• The bias is added after the scores are scaled; adding it before divides the learned signal by the square root of the head size.
• The offset is key minus query; reversed, the diagonal lands in bucket 16, which never trained, and the vectors still look valid.
• The bucketing uses the reference's defaults, 32 buckets and a distance of 128, not the config; candding refuses any other bucket count.
• The bias comes from plain indices 0..seq, not from the position ids, which collapse to the padding index in a padded row.
• There is no token type table; a config struct borrowed from BERT would default one into existence.
• The epsilon is 1e-5, and the reference config class defaults it to 1e-12; read it from the checkpoint.
• The conflicting do_lower_case flags are harmless: both implementations lowercase once, through tokenizer.json.
• pooler.dense.* and the embeddings.position_ids buffer are in the checkpoint and never run.

References