Qwen3

Qwen3-Embedding turns the Qwen3 decoder into an embedder: the text is read left to right, the hidden state of the final token is the vector, and an instruction in front of a query steers what "similar" means. Three sizes share one implementation; the same backbone also runs three rerankers and a sparse head.

What it is

A decoder-only transformer with causal attention, rotary positions at a base of one million, RMSNorm before each sublayer at an epsilon of 1e-6, a SwiGLU feed-forward, grouped-query attention with eight key-value heads, and per-head RMSNorms on the queries and keys before the rotation. The byte-level BPE tokenizer appends the end-of-sequence token to every input, and that token's hidden state after the final norm is the embedding.

How candding implements it

The loader takes bare tensor names or a causal language model's model. prefix, and a config.json with is_causal: false gets a bidirectional padding mask instead.

Pooling and templates

Pooling is last_token. Queries carry an instruction, passages nothing, so one passage index serves every task.

The default instruction is Given a web search query, retrieve relevant passages that answer the query, which query_embed_with_instruction replaces for one call; the rerankers render the same default in their prompt. Every size supports Matryoshka truncation, documented from 32 dimensions up to the full size.

The default length is 8192 tokens. The builder's max_length raises it to the position limit, 32768 for Qwen3-Embedding-0.6B and 40960 for the other sizes and every reranker, at an attention memory quadratic in the length.

Dtypes and memory

The checkpoints are stored in BF16. The 0.6B models load in F32 by default, and the 4B and 8B in BF16, which the CPU turns into F32. F16 is refused on the CPU for the 0.6B and 4B embedding models, since candle accumulates a CPU F16 matmul in F16 and the 0.6B misses its tolerance there; the 8B's CPU F16 is untested and loads with a warning.

The rerank page has the rerankers' cells.

Supported models

Pitfalls

• The query prompt has no space after Query:; adding one changes the tokens and the vector.
• The end-of-sequence token comes from the tokenizer's post-processor, so token_count of an empty string is 1; a port that appends the token itself doubles it.
• The query and key norms apply per head, after the projection and before the rotation.
• The rotary tables are computed in F32 and cast to the model dtype; tables built directly in F16 lose positions past a few thousand.
• config.json has no pad_token_id; the padding id comes from tokenizer_config.json's pad_token.
• The reranker prompt ends in </think> and two newlines, where the score is read; chat_template.jinja ends in three.
• A reranker prompt cut from the right loses the suffix the score is read at unless the suffix is written back.
• The reranker score reads the hidden state after the final norm, not before it.
• A local dense Qwen3 directory whose config.json sets is_causal: false runs bidirectionally but is still last-token pooled, with no warning.

References