candding
Models

GTE (new)

The GTE (new) encoder behind Alibaba-NLP/gte-multilingual-base and gte-base-en-v1.5: rotary positions with NTK scaling, CLS pooling, 8192 tokens.

GTE (new) is the encoder both Alibaba-NLP/gte-multilingual-base (mGTE) and Alibaba-NLP/gte-base-en-v1.5 (en-v1.5) declare as "model_type": "new". It is a third architecture under the GTE name: thenlper/gte-base and -large run on BERT, and Alibaba-NLP/gte-modernbert-base on ModernBERT.

What it is

An encoder-only transformer with bidirectional attention and no position table, only a rotary table built at load. Both checkpoints have twelve layers of hidden size 768, twelve heads, the erf GELU, LayerNorm at 1e-12, one fused query, key and value projection and post-norm blocks with no final norm; they differ in almost everything else:

mGTEen-v1.5
TokenizerSentencePiece Unigram, <s> A </s>lowercasing WordPiece, [CLS] A [SEP]
Token type tableone rownone
Tensor prefixnew.none
rope_theta, rope_scaling.factor20000, 8500000, 2
modules.jsonTransformer, Pooling, NormalizeTransformer, Pooling
WeightsF16F32

How candding implements it

The rotary table is built at rope_theta * factor, with every frequency also scaled by factor^(-2/head_dim), and the tensor prefix is probed at load. candding refuses any other hidden_act, layer_norm_type or position_embedding_type, pack_qkv, unpad_inputs, memory-efficient attention, a rope_scaling.type other than ntk, and type_vocab_size > 1.

mGTE's checkpoint also carries a token-classification head, which candding discards; scoring it as sparse weights is on the roadmap.

Pooling and templates

Both checkpoints pool cls and carry no templates and no instruction placeholder; their default length is 8192. mGTE supports Matryoshka truncation, from 128 to 768 by its model card; en-v1.5 claims none. en-v1.5 ships no Normalize module, which the pooling and templates page explains.

Supported models

ModelFamilyDimPoolingMax lengthDtypesCPUMetalCUDALicense
Alibaba-NLP/gte-multilingual-baseGTE (new)768cls8192f32f16bf16verifiedverifiedverifiedApache-2.0
Alibaba-NLP/gte-base-en-v1.5GTE (new)768cls8192f32f16bf16verifiedverifiedverifiedApache-2.0

Dtypes: bold is the default the builder loads; plain text is verified, muted is untested, struck through is unsupported, each colored by the dtype's own Metal status so a processor-only refusal does not read as unsupported everywhere; hover a dtype for its own cpu/metal/cuda breakdown. BF16 is refused on the CPU for every row: candle has no CPU BF16 matmul.

Pitfalls

  • The two halves of up_gate_proj have the same shape, so swapping them still runs; the GELU takes the second half here, the first in ModernBERT.
  • rope_scaling applies at every length, not past a trained length as NomicBERT's dynamic NTK does; read the NomicBERT way, it builds a plain rope_theta table.
  • en-v1.5's model card calls the scaling equivalent to multiplying the base by the factor; that misses the frequency scale, and the vectors move.
  • type_vocab_size is required: 0 means no table and 1 a one-row table, so no default is safe.
  • A fixed tensor prefix fails to load one of the two checkpoints, since only mGTE's carries new..
  • logn_attention_scale and logn_attention_clip1 are inert: the reference's code that would apply them is commented out.
  • en-v1.5's tokenizer.json declares truncation and fixed padding at 512 tokens; candding's loader replaces both, so the 8192 default applies.

References

Li et al. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv:2308.03281
Zhang et al. (2024). mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. arXiv:2407.19669
Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864
Shazeer (2020). GLU Variants Improve Transformer. arXiv:2002.05202

On this page