ModernBERT
The ModernBERT encoder behind Alibaba-NLP/gte-modernbert-base: alternating global and local attention, two rotary bases, CLS pooling, 8192 tokens.
ModernBERT replaces BERT's position table with two rotary embeddings, alternates layers between global attention and attention inside a local window, and uses a GeGLU feed-forward with no biases anywhere. The catalog's one ModernBERT checkpoint is Alibaba-NLP/gte-modernbert-base.
What it is
An encoder-only transformer with bidirectional attention and pre-norm blocks, h = h + attn(attn_norm(h)) then h = h + mlp(mlp_norm(h)), then one final_norm and no pooler.
| Layers | Hidden | Heads | Head size | Intermediate | Vocabulary | Local window | Rotary base, global and local |
|---|---|---|---|---|---|---|---|
| 22 | 768 | 12 | 64 | 1152 | 50368 | 128 | 160000, 10000 |
Layer i attends globally when i % 3 == 0, layer 0 included, and inside the local window otherwise. Attention reads one fused Wqkv projection, no tensor carries a bias, and every LayerNorm is weight-only. The default length is 8192, the checkpoint's max_position_embeddings, since the repository ships no sentence_bert_config.json.
How candding implements it
Global and local layers rotate at their own bases, and the feed-forward is Wo(gelu(first) * second) over the halves of Wi's output, with the erf GELU. Layer 0 has no attn_norm weight, so candding uses the identity there, as the reference does. The residual stream peaks within a fifth of F16's largest finite value, so F16 runs close to overflow.
Pooling and templates
Pooling is cls, from 1_Pooling/config.json. There are no templates, no instruction placeholder, no Matryoshka support and no Normalize module. The tokenizer is a byte-level BPE, and the family ignores token type ids, since it has no token type table.
Supported models
| Model | Family | Dim | Pooling | Max length | Dtypes | CPU | Metal | CUDA | License |
|---|---|---|---|---|---|---|---|---|---|
| Alibaba-NLP/gte-modernbert-base | ModernBERT | 768 | cls | 8192 | f32f16bf16 | verified | verified | verified | Apache-2.0 |
Dtypes: bold is the default the builder loads; plain text is verified, muted is untested, struck through is unsupported, each colored by the dtype's own Metal status so a processor-only refusal does not read as unsupported everywhere; hover a dtype for its own cpu/metal/cuda breakdown. BF16 is refused on the CPU for every row: candle has no CPU BF16 matmul.
Pitfalls
- The window is
local_attention / 2 + 1as an exclusive bound. Using 64, 128 or the rawlocal_attentionbuilds a finite band of the wrong width. - The reference carries both
config.sliding_window, 64, and the attention module'ssliding_window, 65; only the first reaches the mask. Wi's first half is the activated one and its second half the raw multiplier, the opposite of GTE (new); a swap still runs.classifier_pooling: "mean"belongs to a classification head this checkpoint does not publish; the pooling iscls.position_embedding_type: "absolute"is inert; ModernBERT has no position table.- A whitespace-only input is five tokens, not an empty pair, because the tokenizer works on bytes.