BERT

BERT is Phase 0, the catalog's first family, and one implementation of it covers every catalog model: bge, MiniLM, gte, mxbai-embed and Snowflake Arctic are all the same encoder with different weights, tokenizers, pooling modes and prefixes.

What it is

An encoder-only transformer: every token attends to every other token, positions come from a learned absolute position embedding rather than from a rotation or a bias, a token type embedding is added on top of the word embedding, and each block applies LayerNorm after the residual addition instead of before the sublayer. Nothing is generated and nothing is masked causally; the output is one hidden state per input token, which pooling then reduces to a single vector.

The catalog uses five shapes of the same architecture.

Three tokenizers cover them, and the vocabulary size is the one number that changes the weight shapes without changing the code.

How candding implements it

The family lives in candding/src/models/bert/, split so that each file holds one part of the architecture.

The details that decide whether the vectors match are all small. The activation is the erf-based GELU, because these configs declare hidden_act: gelu and Hugging Face maps that name to the exact form rather than to the tanh approximation. The LayerNorm epsilon is read from the config, 1e-12 for every catalog model, and never hardcoded. The attention scale is 1/sqrt(head_dim), applied to the scores before the mask is added. The additive mask is built once per forward pass in the dtype the hidden states are in, and it is skipped entirely when no row in the chunk is padded. Each block adds the residual and then applies LayerNorm, once after attention and once after the MLP. Weight names follow the Hugging Face checkpoints, with a bert. prefix fallback so that both a bare encoder checkpoint and a full model checkpoint load, and the attention call itself is the shared function described on the devices and dtypes page, so the family code has no backend branch of its own.

Pooling and templates

Two groups run through the same pipeline: the bge-style models take the [CLS] state and prepend a retrieval instruction to queries, and the sentence-transformers-style models average the real tokens and prepend nothing. The passage template is empty for every catalog model, so embed and passage_embed return the same vectors and only query_embed differs.

The lengths are the max_seq_length each repository publishes, and the builder can raise one up to the model's position limit. No catalog model carries an instruction placeholder or declares Matryoshka support; pooling and templates covers both mechanisms and the families that use them.

Supported models

Pitfalls

• gelu in a Hugging Face config is the erf-based GELU; gelu_new and gelu_pytorch_tanh are the tanh approximation, and swapping them shifts every hidden state by a small amount that survives pooling.
• Token type embeddings are added even when every token type id is zero, which is the case for every single-sequence input; skipping the lookup silently changes the result.
• Mean pooling divides by the mask sum clamped at 1e-9, in the dtype the hidden states are in, exactly as sentence-transformers does it.
• The additive mask floor is the most negative finite value of the dtype, not -inf, so a row that is entirely padding softmaxes to a finite vector instead of NaN.
• sentence-transformers rebuilds the tokenizer with a Lowercase normalizer when sentence_bert_config.json says do_lower_case: true, which for BAAI/bge-small-zh-v1.5 contradicts the published tokenizer.json (lowercase: false); candding replays tokenizer.json, as FlagEmbedding, fastembed and text-embeddings-inference do, so the reference generator pins the normalizer to that file and capitalized Latin text in that one model is a deliberate divergence from a plain sentence-transformers run.
• thenlper/gte-base, thenlper/gte-large and mixedbread-ai/mxbai-embed-large-v1 declare torch_dtype: float16 in config.json, so the reference generator forces F32 parameters and asserts them; without that the reference would be an fp16 forward pass compared against an F32 one.
• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 pads with id 0 because that is its pad_token_id, although its <pad> token is id 1. It is harmless: padded positions are masked in attention and excluded from mean pooling, so the padding id never reaches the output.

References