GTE (new)
The GTE (new) encoder behind Alibaba-NLP/gte-multilingual-base and gte-base-en-v1.5: rotary positions with NTK scaling, CLS pooling, 8192 tokens.
GTE (new) is the encoder both Alibaba-NLP/gte-multilingual-base (mGTE) and Alibaba-NLP/gte-base-en-v1.5 (en-v1.5) declare as "model_type": "new". It is a third architecture under the GTE name: thenlper/gte-base and -large run on BERT, and Alibaba-NLP/gte-modernbert-base on ModernBERT.
What it is
An encoder-only transformer with bidirectional attention and no position table, only a rotary table built at load. Both checkpoints have twelve layers of hidden size 768, twelve heads, the erf GELU, LayerNorm at 1e-12, one fused query, key and value projection and post-norm blocks with no final norm; they differ in almost everything else:
| mGTE | en-v1.5 | |
|---|---|---|
| Tokenizer | SentencePiece Unigram, <s> A </s> | lowercasing WordPiece, [CLS] A [SEP] |
| Token type table | one row | none |
| Tensor prefix | new. | none |
rope_theta, rope_scaling.factor | 20000, 8 | 500000, 2 |
modules.json | Transformer, Pooling, Normalize | Transformer, Pooling |
| Weights | F16 | F32 |
How candding implements it
The rotary table is built at rope_theta * factor, with every frequency also scaled by factor^(-2/head_dim), and the tensor prefix is probed at load. candding refuses any other hidden_act, layer_norm_type or position_embedding_type, pack_qkv, unpad_inputs, memory-efficient attention, a rope_scaling.type other than ntk, and type_vocab_size > 1.
mGTE's checkpoint also carries a token-classification head, which candding discards; scoring it as sparse weights is on the roadmap.
Pooling and templates
Both checkpoints pool cls and carry no templates and no instruction placeholder; their default length is 8192. mGTE supports Matryoshka truncation, from 128 to 768 by its model card; en-v1.5 claims none. en-v1.5 ships no Normalize module, which the pooling and templates page explains.
Supported models
| Model | Family | Dim | Pooling | Max length | Dtypes | CPU | Metal | CUDA | License |
|---|---|---|---|---|---|---|---|---|---|
| Alibaba-NLP/gte-multilingual-base | GTE (new) | 768 | cls | 8192 | f32f16bf16 | verified | verified | verified | Apache-2.0 |
| Alibaba-NLP/gte-base-en-v1.5 | GTE (new) | 768 | cls | 8192 | f32f16bf16 | verified | verified | verified | Apache-2.0 |
Dtypes: bold is the default the builder loads; plain text is verified, muted is untested, struck through is unsupported, each colored by the dtype's own Metal status so a processor-only refusal does not read as unsupported everywhere; hover a dtype for its own cpu/metal/cuda breakdown. BF16 is refused on the CPU for every row: candle has no CPU BF16 matmul.
Pitfalls
- The two halves of
up_gate_projhave the same shape, so swapping them still runs; the GELU takes the second half here, the first in ModernBERT. rope_scalingapplies at every length, not past a trained length as NomicBERT's dynamic NTK does; read the NomicBERT way, it builds a plainrope_thetatable.- en-v1.5's model card calls the scaling equivalent to multiplying the base by the factor; that misses the frequency scale, and the vectors move.
type_vocab_sizeis required: 0 means no table and 1 a one-row table, so no default is safe.- A fixed tensor prefix fails to load one of the two checkpoints, since only mGTE's carries
new.. logn_attention_scaleandlogn_attention_clip1are inert: the reference's code that would apply them is commented out.- en-v1.5's
tokenizer.jsondeclares truncation and fixed padding at 512 tokens; candding's loader replaces both, so the 8192 default applies.
References
ModernBERT
The ModernBERT encoder behind Alibaba-NLP/gte-modernbert-base: alternating global and local attention, two rotary bases, CLS pooling, 8192 tokens.
XLM-RoBERTa (flash)
The XLM-RoBERTa (flash) encoder behind jina-colbert-v2 and jina-reranker-v2-base-multilingual, with rotary or learned positions per checkpoint.