candding
Models

ModernBERT

The ModernBERT encoder behind Alibaba-NLP/gte-modernbert-base: alternating global and local attention, two rotary bases, CLS pooling, 8192 tokens.

ModernBERT replaces BERT's position table with two rotary embeddings, alternates layers between global attention and attention inside a local window, and uses a GeGLU feed-forward with no biases anywhere. The catalog's one ModernBERT checkpoint is Alibaba-NLP/gte-modernbert-base.

What it is

An encoder-only transformer with bidirectional attention and pre-norm blocks, h = h + attn(attn_norm(h)) then h = h + mlp(mlp_norm(h)), then one final_norm and no pooler.

LayersHiddenHeadsHead sizeIntermediateVocabularyLocal windowRotary base, global and local
227681264115250368128160000, 10000

Layer i attends globally when i % 3 == 0, layer 0 included, and inside the local window otherwise. Attention reads one fused Wqkv projection, no tensor carries a bias, and every LayerNorm is weight-only. The default length is 8192, the checkpoint's max_position_embeddings, since the repository ships no sentence_bert_config.json.

How candding implements it

Global and local layers rotate at their own bases, and the feed-forward is Wo(gelu(first) * second) over the halves of Wi's output, with the erf GELU. Layer 0 has no attn_norm weight, so candding uses the identity there, as the reference does. The residual stream peaks within a fifth of F16's largest finite value, so F16 runs close to overflow.

Pooling and templates

Pooling is cls, from 1_Pooling/config.json. There are no templates, no instruction placeholder, no Matryoshka support and no Normalize module. The tokenizer is a byte-level BPE, and the family ignores token type ids, since it has no token type table.

Supported models

ModelFamilyDimPoolingMax lengthDtypesCPUMetalCUDALicense
Alibaba-NLP/gte-modernbert-baseModernBERT768cls8192f32f16bf16verifiedverifiedverifiedApache-2.0

Dtypes: bold is the default the builder loads; plain text is verified, muted is untested, struck through is unsupported, each colored by the dtype's own Metal status so a processor-only refusal does not read as unsupported everywhere; hover a dtype for its own cpu/metal/cuda breakdown. BF16 is refused on the CPU for every row: candle has no CPU BF16 matmul.

Pitfalls

  • The window is local_attention / 2 + 1 as an exclusive bound. Using 64, 128 or the raw local_attention builds a finite band of the wrong width.
  • The reference carries both config.sliding_window, 64, and the attention module's sliding_window, 65; only the first reaches the mask.
  • Wi's first half is the activated one and its second half the raw multiplier, the opposite of GTE (new); a swap still runs.
  • classifier_pooling: "mean" belongs to a classification head this checkpoint does not publish; the pooling is cls.
  • position_embedding_type: "absolute" is inert; ModernBERT has no position table.
  • A whitespace-only input is five tokens, not an empty pair, because the tokenizer works on bytes.

References

Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663
Zhang et al. (2024). mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. arXiv:2407.19669
Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864
Shazeer (2020). GLU Variants Improve Transformer. arXiv:2002.05202

On this page