Pipeline

Stages

One call to embed, query_embed or passage_embed runs the same nine steps for every model and every backend.

1. Template. template::apply takes the descriptor's query or passage template and replaces the {text} and {instruction} placeholders. A template of None returns the text unchanged, which is what embed always does, and placeholders that appear in the text itself are left alone because the substitution runs once.
2. Tokenizer. The published tokenizer.json is replayed as it is, with right padding for every model and truncation to the effective max length counting special tokens.
3. Padded batch. Input ids, the attention mask and token type ids leave the tokenizer as (batch, seq) tensors of u32 on the target device, where seq is the longest row in the chunk, plus one real-token length per row taken from the mask.
4. Family forward. The Encoder implementation for the architecture returns per-token hidden states, (batch, seq, hidden).
5. Pooling. CLS, mean or last token reduces that to one vector per row, (batch, hidden).
6. Upcast. The pooled tensor is converted to F32, whatever dtype the weights were loaded in.
7. Matryoshka truncation. The *_dim methods keep the first dim columns, (batch, dim). A model whose descriptor does not allow it never reaches this step.
8. Normalization. When the descriptor says the model normalizes, each row is divided by its L2 norm, with the norm clamped at 1e-12 so a zero row cannot divide by zero.
9. Order restore. Each row becomes a Vec<f32> and the rows are moved back to the caller's order, one vector per input text.

Batching

The last argument of every embedding method is the batch size: None uses the builder default of 32, and any value you pass overrides it for that call. Before chunking, inputs are sorted by token length in descending order, with ties broken by the original index so the order is deterministic. Each chunk is padded to its own longest row, so keeping similar lengths together is what keeps the padding down.

Five texts at a batch size of two sort and chunk like this.

Input 2 is two tokens long and is padded to two columns rather than sharing a chunk with the 512-token input 1. The output is still ordered 0, 1, 2, 3, 4: the sort is an internal detail, and an empty input list returns an empty vector without touching the model.

Effective max length

The tokenizer's limit is the builder's max_length when you set one, otherwise the descriptor's default, and it is capped at the model's max_position_embeddings in both cases. max_length() reports the value that was chosen.

A repository outside the registry that brings no descriptor gets its own max_position_embeddings capped at 512. The limit counts special tokens, so a BERT tokenizer that wraps the text in [CLS] and [SEP] fits 510 content tokens under a limit of 512, and token_count reports the number of tokens that survive truncation rather than the number the text would need.

Where families plug in

Each architecture is one implementation of one trait and nothing else. Pooling, templates, truncation and normalization live outside family code, so they are written once and behave identically for every model.

pub struct EncoderInput<'a> {
  pub input_ids: &'a Tensor,
  pub attention_mask: &'a Tensor,
  pub token_type_ids: &'a Tensor,
  pub lengths: &'a [usize],
}

pub trait Encoder: Send + Sync {
  fn forward(&self, input: &EncoderInput) -> Result<Tensor>;
  fn hidden_size(&self) -> usize;
  fn max_position_embeddings(&self) -> usize;
}

The model_type field of config.json selects the implementation when the model loads, which is why an unregistered repository with a supported architecture works without a registry entry. Family code contains no backend branches at all; the one place a backend is checked is the shared attention function described on the devices and dtypes page. Adding a family is a checklist rather than a design exercise, and the BERT family page is the worked example.