Gemma3

EmbeddingGemma is the Gemma 3 text model turned into an encoder: attention runs in both directions, the hidden states of all real tokens are averaged, and two Dense layers reshape the result before it is normalized. One size ships, google/embeddinggemma-300m, and its repository is gated.

What it is

A decoder-shaped transformer used bidirectionally. Twenty-four layers alternate between local and global attention in a five-to-one pattern: five layers see a sliding window around each position, the sixth sees the whole input, and the pattern repeats four times. The checkpoint stores that window as 512, but the reference halves it and adds one for bidirectional attention, so a position actually attends within |i - j| < 257. Each layer has three query heads sharing one key-value head, a head size of 256, a query and key RMSNorm per head, a rotary embedding whose base differs between local and global layers, four RMSNorms (before and after attention, before and after the MLP), and a gated MLP with the tanh GELU. Every RMSNorm scales by one plus its weight rather than by the weight itself, and the token embedding is multiplied by the square root of the hidden size before the first layer.

The tokenizer is the Gemma SentencePiece vocabulary of about 262 thousand entries, wrapping every input in a beginning and an end token, so token_count of an empty string is 2. The maximum length is 2048 tokens.

How candding implements it

The family lives in candding/src/models/gemma3/ and assembles the shared layers.

The masks are the difference from the decoder families: global layers get the padding mask, local layers get the sliding mask, in which a position attends to another when their distance is below the window and the key is a real token; both are built once per forward and shared by every layer of their kind. The rotation uses a base of ten thousand on local layers and one million on global layers. The 1 + w norm convention is a load-time transformation in the shared RMSNorm, and the embedding scale is a tensor in the model dtype, so a half-precision run rounds it the way the reference does.

After pooling, the two Dense modules from the repository (2_Dense, 768 to 3072, and 3_Dense, 3072 to 768, both without bias) run in F32, then Matryoshka truncation and normalization; the pipeline page lists the stages.

Pooling, templates and instructions

Pooling is mean over the real tokens, prompt included. Queries carry a task instruction, passages a fixed prefix.

The default instruction is search result. The model card and the repository's prompt file define the other tasks, reached through query_embed_with_instruction.

Matryoshka truncation is supported: the model documents 768, 512, 256 and 128 as its sizes, and candding accepts any prefix length from 1 to 768 and re-normalizes it, leaving choices outside the documented sizes to the caller.

Dtypes and memory

The checkpoint is stored in F32 and loads in F32 by default; the weights are about 1.2 GB in F32 and 0.6 GB in BF16. F32 matches the sentence-transformers reference on CPU and on Metal, and BF16 matches it on Metal. BF16 has no CPU run because candle has no CPU BF16 matmul: an explicit request there is refused and a BF16 default would fall back to F32, as on every model.

F16 is refused on every backend when the model is asked for by its catalog id: .dtype(DType::F16) then returns CanddingError::UnsupportedDtype instead of loading. The refusal reads a catalog entry, so a snapshot loaded from a local directory has none to read and would produce the NaN vectors described below; a descriptor supplied by the caller is read in the entry's place, and refuses or does not according to what it says. The residual stream reaches about 60 000 by the twenty-third layer, and the last layer carries it past the 65 504 an F16 value can hold, so the output would be NaN regardless of device; transformers 5.16.1 in float16 overflows the same way, with peaks within 0.3 % of candding's, which is why the limit is recorded against the model rather than worked around. The catalog table below carries the verification status per dtype and backend.

The repository is gated: accept the license on Hugging Face and log in with hf auth login before the first download, otherwise the builder reports CanddingError::GatedModel. Because of the gate the model is not part of CI; the golden and parity suites run locally and on Metal.

Supported models

Pitfalls

• Every RMSNorm multiplies by 1 + w, not w; loading the weights as plain scales produces vectors that look normal and are wrong everywhere.
• The embedding is scaled by the square root of the hidden size, and the reference casts that scalar to the model dtype before multiplying, so a BF16 run uses the rounded value.
• The sliding window is exclusive and applies in both directions. The checkpoint stores it as 512, but the reference halves it and adds one for bidirectional attention, so a position actually attends within |i - j| < 257; the long fixture is longer than that effective window, so the golden suite exercises it.
• Local and global layers use different rotary bases; sharing one table shifts every global layer.
• The attention scale is query_pre_attn_scalar^-0.5, which equals the head size here but is read from the config, not derived.
• The Dense modules run after pooling and before Matryoshka truncation, in modules.json order, on the F32 pooled vector; putting them before pooling or after truncation changes the result.
• The tokenizer adds both the beginning and the end token; the mean includes both and the prompt.
• The sentence-transformers prompt names (Retrieval-query, Clustering, STS, …) are names for the instructions above; candding takes the instruction text.

References