Qwen3

Qwen3-Embedding turns the Qwen3 decoder into an embedder: the text is read left to right, the hidden state of the final token is the vector, and an instruction placed in front of a query steers what "similar" means. Three sizes share one implementation and one prompt.

What it is

A decoder-only transformer: causal attention, so each token only sees the tokens before it; rotary position embeddings instead of a learned position table; RMSNorm before each sublayer rather than LayerNorm after it; a gated MLP with the SiLU activation; grouped-query attention with eight key-value heads shared by the query heads; and a per-head RMSNorm on the queries and keys before the rotation. There is no token type embedding and no language-model head in the embedding checkpoints. The tokenizer appends the end-of-sequence token to every input, so the last real token is always that marker, and its hidden state after the final norm is the embedding.

All three use a rotary base of one million, an RMSNorm epsilon of 1e-6, and a byte-level BPE tokenizer with NFC normalization and about 151 thousand entries.

How candding implements it

The family lives in candding/src/models/qwen3/ and assembles shared layers rather than defining its own math.

The pieces the family shares with the rest of the crate are the ones that decide correctness. layers::rms_norm computes in F32 and returns the input dtype, as transformers does. layers::rope builds the cosine and sine tables once per model from rope_theta and applies the rotate-half convention through candle's kernel; positions start at zero for every row, and because padding is on the right that is where the reference puts them too. layers::mlp is the SwiGLU block. The attention call is the shared function described on the devices and dtypes page: on Metal candle's kernel handles the grouped heads and the causal flag itself, elsewhere the key and value heads are tiled and a triangular additive mask is added. The causal mask carries no padding term, because a real token can never attend to a padding position that comes after it, and the padded positions are discarded by pooling.

The checkpoints carry bare tensor names, embed_tokens, layers.N and norm, without the model. prefix of a full causal language model; the loader accepts both. The 8B checkpoint ships an untied language-model head that is never read.

Pooling, templates and instructions

Pooling is last_token: the hidden state at index length - 1 of each row, which under right padding is the appended end-of-sequence token. Queries carry an instruction, passages carry nothing.

Query: is followed by the text with no space, exactly as the model card and the repository's prompt file write it. The default instruction is Given a web search query, retrieve relevant passages that answer the query; query_embed_with_instruction replaces it for one call, for example with Given a question, retrieve Wikipedia passages that answer the question or a code-search instruction. The instruction is part of the query side only; passages are embedded without one so that one passage index serves every task.

Every size supports Matryoshka truncation: embed_dim, query_embed_dim and passage_embed_dim keep a prefix of the vector and re-normalize it, and any dimension from 32 up to the full size is accepted.

The default maximum length is 8192 tokens. The models accept longer inputs, 32768 for the small size and 40960 for the two large ones, and the builder's max_length raises the limit up to that ceiling; attention memory grows with the square of the length, which is why the default stops earlier.

Dtypes and memory

The checkpoints are stored in BF16. The 0.6B model loads in F32 by default; the 4B and 8B models load in BF16 by default, because their F32 weights alone are 16 and 32 gigabytes. .dtype(...) on the builder overrides the default, with one rule from candle itself: BF16 has no CPU matmul, so an explicit BF16 request on CPU is refused with CanddingError::UnsupportedOnDevice and a BF16 default on CPU falls back to F32 with a warning. F16 runs everywhere, slowly on CPU.

The catalog table below carries the verification status per dtype; the testing page explains the tolerance column each dtype is held to and why the 8B reference was produced in BF16.

Supported models

Pitfalls

• The query prompt has no space after Query:; adding one changes the tokens and the vector.
• The end-of-sequence token comes from the tokenizer's post-processor, not from the model code; token_count on an empty string returns 1, and a port that appends the token itself would double it.
• Last-token pooling must read the row length from the attention mask; the final column of a right-padded batch is padding for every row but the longest.
• The query and key norms apply per head, over the head size, after the projection and before the rotation; normalizing the whole projection or rotating first gives plausible but wrong vectors.
• The rotary tables are computed in F32 and cast to the model dtype, as transformers casts its cosine and sine to the activation dtype; building them directly in F16 loses positions past a few thousand.
• The causal mask on the matmul path uses the dtype's most negative finite value, not negative infinity, so a fully masked row stays finite.
• config.json has no pad_token_id; the padding id comes from the pad_token named in tokenizer_config.json, and because padded positions never reach the output it only has to be a valid id.
• Loading a BF16 checkpoint as F32 is lossless; loading it as F16 rounds every weight, and the F16 run is held to the half-precision tolerance column like BF16.

References