GTE (new)

Both catalog checkpoints declare "model_type": "new" — the bare English word NewConfig.model_type hardcodes in the shared Alibaba-NLP/new-impl code that both pin their weights to by auto_map — a model_type distinct from the two other families "GTE" already names in this catalog: thenlper/gte-base and -large are Family::Bert, and Alibaba-NLP/gte-modernbert-base is Family::ModernBert; this family is the third. Alibaba-NLP/gte-multilingual-base (mGTE) and Alibaba-NLP/gte-base-en-v1.5 (en-v1.5) share one encoder implementation and disagree on nearly everything else: a new. tensor prefix on one and none on the other, a one-row token-type table on one and none on the other, two different rotary bases, two unrelated tokenizers, and a Normalize pipeline step on one and not the other. Both are Apache-2.0, ungated and 768-dimensional; mGTE is 0.61 GB on disk in F16, en-v1.5 is 0.55 GB in F32.

What it is

An encoder-only transformer with bidirectional attention, no position table of any kind, and every positional signal carried by a rotary table built once at load time. Both checkpoints share the same twelve-layer, 768-hidden, twelve-head, 3072-intermediate shape, hidden_act: gelu (the erf form), layer_norm_type: layer_norm with eps: 1e-12, position_embedding_type: rope, a fused pack_qkv: true projection, and max_position_embeddings: 8192, matched by both repositories' own sentence_bert_config.json max_seq_length. GteNewConfig::ensure_supported refuses every other value hidden_act, layer_norm_type, position_embedding_type and pack_qkv can carry, along with unpad_inputs: true, use_memory_efficient_attention: true, any rope_scaling.type besides "ntk" and type_vocab_size above 1 — combinations neither checkpoint exercises; layer_norm_eps and max_position_embeddings are read as published, with no check on either.

The tensor prefix and the token-type table

GteNewModel::load probes for new.embeddings.word_embeddings.weight before falling back to the bare root, because mGTE's checkpoint carries a new. prefix on 136 of its 138 tensors and en-v1.5's 135 carry none — NewPreTrainedModel.base_model_prefix = "new" showing through because the published mGTE repository is a NewForTokenClassification wrapper around the backbone, whose constructor names the field self.new. The two tensors without the prefix are that same wrapper's own token-classification head, classifier.weight [1, 768] and classifier.bias [1], which NewModel and this port both discard on load; scoring them into per-token sparse weights, the way the repository's own scripts/gte_embedding.py does, is Phase 3's sparse work, not this family's. GteNewConfig::type_vocab_size has no serde default, because 0 (en-v1.5) and 1 (mGTE) are both real, checkpoint-declared values: 0 means no token-type table exists at all, not a table of size zero, and a struct that defaulted it would build a table a checkpoint omitting the key never published. GteNewEmbeddings::forward adds row 0 of that table to every position whenever it exists — the reference indexes token_type_ids but zeroes that tensor in place first whenever type_vocab_size < 2, so every position reads the same row regardless of what a caller passes, and candding skips straight to that row rather than round-tripping a caller-supplied id tensor through an index that always resolves to zero.

The rotary table

Both checkpoints take the reference's NTK-scaling branch of its rotary constructor, which rescales unconditionally at construction rather than past a trained length — this family's ordinary table, not a length-gated branch the way NomicBERT's dynamic NTK is. GteNewConfig::rope_theta_effective returns rope_theta * rope_scaling.factor (160000 for mGTE, 1000000 for en-v1.5), and GteNewConfig::rope_frequency_scale returns the uniform constant factor^(-2/head_dim) (0.937084 for mGTE, 0.978572 for en-v1.5) that scales every entry of the table built at that effective theta — a prefactor constant across the head dimension rather than a second geometric term, which no single theta argument to RotaryEmbedding::new can express. RotaryEmbedding::with_frequency_scale carries the extra multiplier, and GteNewModel::load is its only caller today.

Two ways of getting this table wrong were measured directly on the reference implementation itself — a wrong table substituted into its own loaded rotary module and compared against the correct one on the nine catalog fixtures, a check of the reference against itself, not of candding:

The second row is en-v1.5's own model card, whose evaluation note reads "set ntk scaling factor to 2 (equivalent to rope_base * 2)": the code does not stop at rope_theta * factor, it also divides every inv_freq entry by factor^(2/head_dim), and the card's simplification is what the second row costs on the reference's own output.

The gated feed-forward

GteNewMlp::load reads one bias-free up_gate_proj of shape (2 * intermediate, hidden) and splits its rows in half: rows 0..intermediate are the raw multiplier (GatedMlp's up) and rows intermediate..2*intermediate are the half the erf GELU runs over (GatedMlp's gate) — the opposite of ModernBERT's Wi, where the first half is activated. A biased down_proj projects back down to hidden; up_gate_proj itself carries no bias.

The post-norm layer

GteNewLayer::forward runs two residuals, each followed by a norm: h = attn_ln(h + attn(h)), then h = mlp_ln(h + mlp(h)) — post-norm, the opposite of ModernBERT's pre-norm block. attn_ln and mlp_ln sit directly under the layer rather than nested inside attention. or mlp., both are nn.LayerNorm with a bias, and so is embeddings.LayerNorm: no LayerNorm anywhere in this family is bias-free, unlike ModernBERT, where none carries one. There is no final norm after the twelfth layer; Encoder::forward returns the last layer's own mlp_ln output directly.

Dtype headroom

Per-layer maximum absolute hidden state, measured in the reference at F32 on each checkpoint's own long fixture:

F16's largest finite value is 65504; the two peaks sit at 0.066% and 0.026% of that ceiling, far below ModernBERT's 81% on the same measurement, because the post-norm block renormalizes twice a layer and the residual stream never accumulates the way a pre-norm one can. The reference's own float16 forward of the same fixtures produced no inf and no NaN on either checkpoint, at a CLS cosine of 0.9999996 (mGTE) and 0.9999989 (en-v1.5) against its own F32 run. Whether candding's own F16 forward stays finite and within tolerance on these checkpoints is what the f16 badges in the table below record, not a fact this page states.

Pooling, normalization and the elastic dimension

Both 1_Pooling/config.json files set pooling_mode_cls_token: true with every other mode false, so both resolve to Pooling::Cls; both sentence_bert_config.json files set max_seq_length: 8192. mGTE's modules.json adds a Normalize step, so its raw pooled output is already unit length, while en-v1.5's stops at Pooling and its own card presents normalization as optional; candding normalizes every registry entry's pooled output regardless of what modules.json declares, the same divergence the ModernBERT and paraphrase-multilingual-mpnet-base-v2 pages already document for their own missing Normalize module. mGTE's modules.json also names a 2_Normalize directory that does not exist anywhere in the repository — harmless, since Normalize holds no weights of its own, but a reminder that this file is not a directory listing.

mrl is true on the multilingual entry alone. Its model card is the only file anywhere in the repository, or in the shared reference code, that names an elastic dimension for it — a comment reading "should be in [128, 768]" next to a slice of the pooled vector — and no config.json, modules.json or sentence_bert_config.json key states any such rule. The card's own transformers snippet that performs that slice is wrong: outputs.last_hidden_state[:, 0][:dimension] indexes the batch axis, not the feature axis, so at dimension=768 it silently returns every row at full width instead of truncating any of them. Only the repository's own scripts/gte_embedding.py, which slices [:, 0, :dimension] correctly, implements the rule as documented, and hub::INCLUDE_PATTERNS has no .py entry, so candding never fetches that file. en-v1.5's own card and files make no elastic-dimension claim at all, which is why its entry carries mrl: false.

The tokenizer

mGTE and en-v1.5 do not share a tokenizer at all — only the file name matches. mGTE's tokenizer.json is a 250002-entry Unigram/SentencePiece model with a Precompiled normalizer and a WhitespaceSplit plus Metaspace pre-tokenizer, producing <s> A </s> for a single input and keeping every segment of a pair at type_id 0; special ids are <s> 0, </s> 2, <pad> 1, <unk> 3, <mask> 250001. en-v1.5's is a 30522-entry WordPiece model with a lowercasing BertNormalizer and a BertPreTokenizer, producing [CLS] A [SEP] for a single input and moving a pair's second segment and its [SEP] to type_id 1; special ids are [CLS] 101, [SEP] 102, [PAD] 0, [UNK] 100, [MASK] 103. config.json's own vocab_size — 250048 for mGTE, 30528 for en-v1.5 — is wider than either tokenizer's own vocabulary by 46 and 6 rows respectively, and neither tokenizer can address those extra embedding rows. Both tokenize an empty string and a whitespace-only string to the same two special ids with nothing in between; the resulting vectors are byte-identical for en-v1.5 (both rows carry norm 20.286856) and match to a cosine of 0.99999994 for mGTE, floating-point noise from the batched forward rather than a difference in the tokens themselves. en-v1.5's own tokenizer.json also bakes in a truncation.max_length of 512 and a Fixed(512) padding declaration, at odds with its own sentence_bert_config.json and card, both of which say 8192; candding's own tokenizer loader installs its own truncation and padding on top of a published file, so both baked-in declarations are overridden and never reached.

Supported models

Pitfalls

Most mistakes below produce a finite, plausible-looking vector rather than an error or a NaN, and need a reference comparison rather than a shape or finiteness check to catch; two of them — the token-type default and the tensor prefix — instead fail loudly with an Err at load, as their own bullets say.

• The up_gate_proj halves swapped. Both halves are the same shape, (intermediate, hidden), so a swap still runs to completion and only the numeric result changes; it is the second half the activation runs over here, the opposite of ModernBERT's Wi.
• rope_scaling read as a past-the-trained-length branch. Treating it the way NomicBERT's dynamic NTK reads its own config key builds a plain, unscaled rope_theta table — well-formed and finite, and it costs cosine 0.974369 on mGTE's long fixture, measured on the reference itself above.
• The card's "equivalent to rope_base * 2" simplification. Building the table at rope_theta * factor alone, without the uniform factor^(-2/head_dim) correction, costs cosine 0.997348 to 0.997935, also measured on the reference above — small enough to look like an acceptable approximation and wrong regardless.
• type_vocab_size defaulted instead of required. NewConfig's own Python default is 1, and a serde default only takes effect when a checkpoint's config.json omits the key — neither published checkpoint does; the mistake would only surface on a future checkpoint that both omits the key and, like en-v1.5, carries no token_type_embeddings tensor, where GteNewEmbeddings::load fails to find it and returns an Err, proven by a_positive_type_vocab_size_fails_to_load_without_a_token_type_tensor, not a silently wrong vector.
• The new. tensor prefix assumed fixed instead of probed. It comes from a token-classification wrapper around mGTE's backbone, not from the architecture itself, and en-v1.5's checkpoint carries no prefix at all; hardcoding either shape leaves every tensor lookup unable to find its name under the wrong root for the other checkpoint, an Err at load rather than a plausible wrong vector, proven by the_prefix_probe_finds_both_shapes's own wrong_prefix case.
• logn_attention_scale and logn_attention_clip1 read as live fields. Both exist in the reference's config class and both are checked in its forward pass, but the lines that would compute the scale are commented out in this revision of the reference, so the key is inert in both published checkpoints regardless of its value.

References