candding
Models

BERT

The BERT encoder family behind bge, MiniLM, gte, mxbai and Snowflake Arctic embeddings, and the details that make its vectors match the reference.

BERT is the encoder behind most of the dense catalog: bge, MiniLM, thenlper's gte, mxbai-embed, Snowflake Arctic and intfloat/multilingual-e5-small are one implementation with different weights, tokenizers, pooling modes and prefixes. The same encoder also serves a sparse, a multi-vector and a rerank output.

What it is

An encoder-only transformer: bidirectional attention, a learned absolute position table, a token type embedding added to the word embedding, and LayerNorm after each residual addition. The catalog's checkpoints use three tokenizers: English WordPiece, Chinese WordPiece for BAAI/bge-small-zh-v1.5, and a multilingual Unigram for paraphrase-multilingual-MiniLM-L12-v2 and multilingual-e5-small.

How candding implements it

The family lives in candding/src/models/bert/, one part of the architecture per file, config.rs, embeddings.rs, attention.rs, layer.rs and model.rs, the layout a new family copies. model.rs loads the weights with or without the bert. prefix.

Other outputCheckpointLoaded throughWhat it adds
sparseprithivida/Splade_PP_en_v1SparseTextEmbeddinga masked-language-model head, max-pooled over the sequence
sparsesentence-transformers/all-MiniLM-L6-v2SparseTextEmbeddingBm42, which reads the last layer's attention weights
multi-vectorcolbert-ir/colbertv2.0, answerdotai/answerai-colbert-small-v1MultiVectorTextEmbeddinga bias-free projection of every surviving position, 768 to 128 and 384 to 96
rerankcross-encoder/ms-marco-MiniLM-L6-v2TextCrossEncoderBertPooler, a linear layer and a tanh over the first token, then a linear layer to one raw logit

Pooling and templates

ModelsPoolingQuery templatePassage template
bge English (small, base, large), mixedbread-ai/mxbai-embed-large-v1, Snowflake Arctic (xs, s, m, l)clsRepresent this sentence for searching relevant passages: {text}none
BAAI/bge-small-zh-v1.5cls为这个句子生成表示以用于检索相关文章：{text}none
thenlper/gte-base, -large, all-MiniLM-L6-v2, paraphrase-multilingual-MiniLM-L12-v2meannonenone
intfloat/multilingual-e5-smallmeanquery: {text}passage: {text}

No BERT model has an instruction placeholder or Matryoshka support. mixedbread-ai/mxbai-embed-large-v1's model card describes Matryoshka truncation, but the checkpoint declares none, so candding does not enable it.

Supported models

ModelFamilyDimPoolingMax lengthDtypesCPUMetalCUDALicense
BAAI/bge-small-en-v1.5BERT384cls512f32f16bf16verifiedverifiedverifiedMIT
BAAI/bge-base-en-v1.5BERT768cls512f32f16bf16verifiedverifiedverifiedMIT
BAAI/bge-large-en-v1.5BERT1024cls512f32f16bf16verifiedverifiedverifiedMIT
BAAI/bge-small-zh-v1.5BERT512cls512f32f16bf16verifiedverifiedverifiedMIT
sentence-transformers/all-MiniLM-L6-v2BERT384mean256f32f16bf16verifiedverifiedverifiedApache-2.0
thenlper/gte-baseBERT768mean512f32f16bf16verifiedverifiedverifiedMIT
thenlper/gte-largeBERT1024mean512f32f16bf16verifiedverifiedverifiedMIT
mixedbread-ai/mxbai-embed-large-v1BERT1024cls512f32f16bf16verifiedverifiedverifiedApache-2.0
Snowflake/snowflake-arctic-embed-xsBERT384cls512f32f16bf16verifiedverifiedverifiedApache-2.0
Snowflake/snowflake-arctic-embed-sBERT384cls512f32f16bf16verifiedverifiedverifiedApache-2.0
Snowflake/snowflake-arctic-embed-mBERT768cls512f32f16bf16verifiedverifiedverifiedApache-2.0
Snowflake/snowflake-arctic-embed-lBERT1024cls512f32f16bf16verifiedverifiedverifiedApache-2.0
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2BERT384mean128f32f16bf16verifiedverifiedverifiedApache-2.0
intfloat/multilingual-e5-smallBERT384mean512f32f16bf16verifiedverifiedverifiedMIT

Dtypes: bold is the default the builder loads; plain text is verified, muted is untested, struck through is unsupported, each colored by the dtype's own Metal status so a processor-only refusal does not read as unsupported everywhere; hover a dtype for its own cpu/metal/cuda breakdown. BF16 is refused on the CPU for every row: candle has no CPU BF16 matmul.

Pitfalls

  • gelu in a Hugging Face config is the erf GELU; gelu_new and gelu_pytorch_tanh are the tanh approximation, and swapping them shifts every hidden state.
  • The token type embedding is added even when every token type id is zero, as for every single-sequence input.
  • The LayerNorm epsilon comes from the config, 1e-12 for these checkpoints and 1e-5 for XLM-RoBERTa and MPNet.
  • Load every LayerNorm through layers::layer_norm::LayerNorm, which keeps its statistics in F32, never through candle_nn::layer_norm directly.
  • BAAI/bge-small-zh-v1.5's sentence_bert_config.json sets do_lower_case: true against its published tokenizer.json; candding replays tokenizer.json, so capitalized Latin text differs from a plain sentence-transformers run.
  • thenlper/gte-base, thenlper/gte-large and mixedbread-ai/mxbai-embed-large-v1 declare torch_dtype: float16, so a reference must force F32 parameters.

References

Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805
Reimers and Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084
Wang et al. (2020). MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. arXiv:2002.10957
Xiao et al. (2023). C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597
Li et al. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv:2308.03281
Merrick et al. (2024). Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models. arXiv:2405.05374

Model catalog

Every model candding ships, generated from the registry: the dense, sparse, multi-vector and rerank catalogs, each row with its own per-device status.

Qwen3

The Qwen3 decoder family: Qwen3-Embedding's instruction prompt and last-token pooling, the rerankers' prompt, and the dtype rules and memory of each size.

On this page

What it isHow candding implements itPooling and templatesSupported modelsPitfalls