Gemma3

EmbeddingGemma is the Gemma 3 text model turned into an encoder: attention runs in both directions, the hidden states of the real tokens are averaged, and two Dense layers reshape the result before it is normalized. One size ships, google/embeddinggemma-300m, and its repository is gated.

What it is

A decoder-shaped transformer run bidirectionally, whose 24 layers alternate five sliding-window layers with one global layer. Each layer has three query heads of size 256 sharing one key-value head, per-head RMSNorms on the queries and keys, four RMSNorms around attention and the gated tanh-GELU feed-forward, and the token embedding is scaled by the square root of the hidden size.

The tokenizer is Gemma's SentencePiece vocabulary of about 262 thousand entries and wraps every input in a beginning and an end token, so token_count of an empty string is 2.

How candding implements it

Global layers take the padding mask and local layers a sliding one, each built once per forward. After pooling, the repository's two bias-free Dense modules, 2_Dense from 768 to 3072 and 3_Dense back to 768, run in F32 before Matryoshka truncation and normalization, as the pipeline lists.

Pooling, templates and instructions

Pooling is mean over the real tokens, prompt and special tokens included.

The default instruction is search result; query_embed_with_instruction takes the others the model card and the repository's prompt file define.

Matryoshka truncation is supported; the model documents 768, 512, 256 and 128 dimensions.

Dtypes and access

The checkpoint is stored in F32, about 1.2 GB, and loads in F32 by default; BF16 halves that. F16 is refused on every device with CanddingError::UnsupportedDtype, because the residual stream passes F16's largest finite value in the last layer and the output would be NaN. A local directory loaded without a descriptor has no catalog entry to refuse from, and returns those NaN vectors.

The repository is gated: accept the licence on Hugging Face and set HF_TOKEN before the first download, or the build fails with CanddingError::GatedModel.

Supported models

Pitfalls

• Every RMSNorm multiplies by 1 + w, not w; plain scales give vectors that look normal and are wrong everywhere.
• The reference casts the embedding scale to the model dtype before multiplying, so a half-precision run uses the rounded value.
• The sliding window is abs(i - j) < 257 in both directions, the stored 512 halved plus one; the long fixture exercises it.
• Local layers rotate at base ten thousand and global ones at one million; sharing one table shifts every global layer.
• The attention scale is query_pre_attn_scalar^-0.5; the scalar equals the head size here, but read it from the config.
• The sentence-transformers prompt names, such as Retrieval-query and STS, name the instructions above; candding takes the instruction text.

References