ModernBERT

ModernBERT replaces BERT's learned position table with two rotary embeddings and alternates every third layer between attending globally and attending inside a local window, layer 0 included. Its feed-forward is GeGLU rather than a plain MLP, every LayerNorm in the checkpoint is weight-only, and the published checkpoint carries no classification or masked-language head. One checkpoint is in the catalog, Alibaba-NLP/gte-modernbert-base: Apache-2.0, ungated, 768-dimensional and 0.30 GB on disk, its weights stored in F16.

What it is

An encoder-only transformer with bidirectional attention and a pre-norm residual block repeated for every layer. global_attn_every_n_layers sets the alternation: layer i attends globally when i % 3 == 0 and inside a local window otherwise, which makes layer 0 a global layer along with 3, 6, 9, 12, 15, 18 and 21 — eight global layers and fourteen sliding ones. Attention reads one fused Wqkv projection rather than three separate ones, and none of the tensors carry a bias: attention_bias, mlp_bias and norm_bias are all false in config.json. The config also carries two fields nothing in the forward pass reads: classifier_pooling: "mean", which belongs to a sequence-classification head this checkpoint does not publish, and position_embedding_type: "absolute", a leftover from a different convention landing in the same key — ModernBERT has no position table of any kind.

default_max_length is 8192, config.json's own max_position_embeddings. The repository ships no sentence_bert_config.json, so sentence-transformers derives max_seq_length from that same figure rather than from a dedicated setting.

The tensors

134 tensors, and not one of them carries a bias: embeddings.tok_embeddings.weight [50368, 768], embeddings.norm.weight [768], final_norm.weight [768], and per layer layers.{i}.attn.Wqkv.weight [2304, 768], attn.Wo.weight [768, 768], mlp_norm.weight [768], mlp.Wi.weight [2304, 768], mlp.Wo.weight [768, 1152]. attn_norm.weight [768] is the one exception: it exists on layers 1 through 21 and not on layer 0, twenty-one of the twenty-two — 3 top-level tensors plus 22 times 5 per-layer tensors plus those 21 attn_norm weights accounts for all 134.

Wqkv's 2304 rows split into query, key and value in that order: rows 0 to 767 are the query for every head, 768 to 1535 the key, 1536 to 2303 the value, the same layout NomicBERT's own fused Wqkv already uses. mlp.Wi's 2304 rows split in half instead, covered under the feed-forward halves below.

The alternating attention

config.json stores the local window as local_attention: 128, and the reference keeps key j for query i when |i - j| <= local_attention / 2, that is |i - j| <= 64, inclusive. AttentionMask::sliding takes an exclusive window instead: it keeps a key when |i - j| < window. The value that reproduces the inclusive rule is therefore local_attention / 2 + 1: 65, not 64 and not 128. ModernBertConfig::attention_window computes exactly that.

Measured directly against the transformers reference at sequence length 300, not computed independently: query position 100 sees 129 keys.

Only the 65-wide exclusive window matches; 64 falls two keys short at each end, and 128 would cover roughly the whole sequence rather than a local band.

The two rope tables

Global layers rotate at a theta of 160000 (global_rope_theta), sliding layers at a theta of 10000 (local_rope_theta), both over the full 64-wide head, in the same rotate-half convention Qwen3 and Gemma3 already use — layers::rope::RotaryEmbedding needs no change, only a second table built at load time. ModernBertModel::load builds both tables once, up to max_position_embeddings, and rope_for(layer) picks the global or the local one by config.is_global(layer) on every forward. Positions run 0 to seq for every row regardless of padding, which right padding keeps correct.

The feed-forward halves

mlp.Wi projects the 768-wide hidden state up to 2304, twice the 1152-wide intermediate size, and the reference chunks its output in half: input, gate = Wi(h).chunk(2), then Wo(gelu(input) * gate) — the first half is the one the activation runs over, the second is the raw multiplier. ModernBertMlp::load narrows Wi.weight's rows the same way, rows 0 to 1151 as the activated gate and 1152 to 2303 as the raw multiplier, and hands both to the shared GatedMlp. The activation is the exact erf GELU, not its tanh approximation: ACT2FN["gelu"](1.0) is 0.8413447141647339 in the reference, matching Tensor::gelu_erf and distinct from the tanh form's 0.8411920070648193.

The pre-norm layer and the final norm

Each block runs two residuals: h = h + attn(attn_norm(h)), then h = h + mlp(mlp_norm(h)). Layer 0's attn_norm is the reference's own nn.Identity() rather than a loaded weight — the checkpoint's safetensors header has no layers.0.attn_norm.weight tensor at all, because the embeddings' own norm already ran immediately before it. ModernBertLayer::load mirrors that: attn_norm is None on layer 0 and a loaded LayerNorm on every other layer. After the twenty-two layers, one final_norm runs once over the whole sequence; there is no pooler in this family.

Every norm in the checkpoint — embeddings.norm, the 21 attn_norm weights, the 22 mlp_norm weights and final_norm, 45 in total — is weight-only, matching norm_bias: false; candding loads all of them through LayerNorm::load_without_bias.

Dtype headroom

Per-layer maximum absolute hidden state, measured in the reference at F32 on the long fixture (950 tokens):

The residual stream climbs sharply between layers 11 and 19, and final_norm collapses it back down to the low tens; a check that only looks at the returned hidden states would see nothing unusual. F16's largest finite value is 65 504, so the measured peak sits at about 81% of that ceiling with no margin to spare. The reference's own float16 forward of the same fixture produced no inf and no NaN, at a CLS cosine of 0.9999993 against its own F32 run and a max absolute difference of 0.01096 on the raw CLS vector. Whether candding's own F16 forward stays finite and within tolerance on this checkpoint is what the f16 badge in the table below records, not a fact this page states.

Pooling and normalization

Pooling is cls, index 0: 1_Pooling/config.json sets pooling_mode_cls_token: true and every other pooling mode false, and the model card's own snippets take last_hidden_state[:, 0] directly. modules.json holds exactly two modules, Transformer and Pooling, with no Normalize step and no *_Dense directory, and config_sentence_transformers.json's prompts field is empty — no query or passage prefix is published for this checkpoint.

sentence-transformers' own encode() does not L2-normalize this model by default, consistent with the missing Normalize module: the nine catalog fixtures come back with L2 norms from about 37.4 to 39.2, not unit length. candding's pipeline normalizes every model's pooled output regardless of what modules.json declares — the same divergence the paraphrase-multilingual-mpnet-base-v2 entry already documents for its own missing Normalize module — so a caller comparing against a raw, unnormalized reference vector needs to normalize that reference by hand first.

The tokenizer

The tokenizer is a byte-level BPE: tokenizer.json's normalizer is NFC only, with no lowercasing and no accent stripping, and its pre-tokenizer is ByteLevel with add_prefix_space: false. Because it operates on bytes rather than discarding whitespace, a whitespace-only input tokenizes to 5 tokens rather than collapsing straight to [CLS] [SEP] the way every WordPiece family in the catalog does; "The quick brown fox jumps over the lazy dog." tokenizes to 12. [CLS] is 50281 and always sits at index 0, [SEP] is 50282, [PAD] is 50283 — the TemplateProcessing post-processor places them the same way on every input. The vocabulary is 50368 entries, and that is also the embedding table's row count exactly: unlike NomicBERT's 30528 rows over a 30522-entry vocabulary, there is no padded remainder here. EncoderInput::token_type_ids is ignored by this family: there is no token-type table, and tokenizer_config.json's own model_input_names does not list token_type_ids either.

Supported models

Pitfalls

Every mistake below produces a finite, plausible-looking vector rather than an error or a NaN, so each needs a reference comparison rather than a shape or finiteness check to catch.

• The sliding window off by a factor of two, or missing the +1. local_attention (128) is the full span and config.sliding_window (64) is already halved; the exclusive argument that reproduces the reference's inclusive rule is local_attention / 2 + 1 = 65. Using 64, 128, or the checkpoint's raw local_attention in place of 65 all build a well-formed, finite band — just the wrong width.
• config.sliding_window (64) and ModernBertAttention.sliding_window (65) swapped. The reference itself carries both numbers in the same forward, and only the first ever reaches the mask that eager and sdpa attention read; reading the second there shifts the window by one on both edges.
• The first half of Wi's output treated as the raw multiplier instead of the activated gate. Both halves are the same shape, so the swap still runs to completion; only the numeric result changes.
• classifier_pooling: "mean" read as this model's own pooling. It belongs to a sequence-classification head the published checkpoint does not carry; 1_Pooling/config.json and the model card's own snippets both pool [CLS].
• Layer 0 given a loaded attn_norm weight instead of the identity. The safetensors header has no layers.0.attn_norm.weight at all; building one anyway invents a weight the checkpoint never published.
• position_embedding_type: "absolute" read as a signal to build a position table. ModernBERT has no position table of any kind; the field is inert, read by nothing in the forward pass.

References