Getting started

Add candding to a Cargo project, embed your first texts, and pick the device and dtype you want.

Add candding to a Cargo project, embed a query and two passages, then choose the device and dtype.

Requirements

Requirement	Needed for
Linux or macOS	every build; Windows is not tested
Rust 1.94 or newer	every build
A C compiler	the default `download` feature, which compiles `aws-lc-sys` from C
Apple Silicon and the `metal` feature	the GPU path on a Mac
An NVIDIA GPU and the `cuda` feature	candle's CUDA backend; BF16 needs `sm_80` or newer, and the catalog shows what is verified on CUDA
Access to the Hugging Face Hub, or a local model directory	a model's first load

Install

cargo add candding

or in Cargo.toml:

[dependencies]
candding = "0.1"

Add features = ["metal"] on Apple Silicon or features = ["cuda"] on NVIDIA; the API page lists every feature.

Your first embedding

candding/examples/quickstart.rs

//! Embed two passages and a query on the auto-selected device and score them.

use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("BAAI/bge-small-en-v1.5").build()?;
  let passages = model.passage_embed(
    &[
      "Embeddings map text to vectors.",
      "Rust is a systems programming language.",
    ],
    None,
  )?;
  let query = model.query_embed(&["what do embeddings do"], None)?;
  for (i, passage) in passages.iter().enumerate() {
    let score: f32 = query[0].iter().zip(passage).map(|(q, p)| q * p).sum();
    println!("passage {i}: {score:.4}");
  }
  Ok(())
}

passage_embed and query_embed apply the model's passage and query templates, and embed applies none. Every vector is a Vec<f32> of length model.dim() and unit length, so a dot product is a cosine similarity; the builder's normalize(false) returns the raw pooled vector instead.

Choosing the device and dtype

candding/examples/devices.rs

//! Pick the device, dtype, max length and batch size explicitly.

use candding::DType;
use candding::TextEmbedding;
use candding::device;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("sentence-transformers/all-MiniLM-L6-v2")
    .device(device::cpu())
    .dtype(DType::F32)
    .max_length(256)
    .batch_size(16)
    .build()?;
  let rows = model.embed(&["hello world"], None)?;
  println!(
    "device {} dim {} max_length {} first {:.4}",
    device::describe(model.device()),
    model.dim(),
    model.max_length(),
    rows[0][0]
  );
  Ok(())
}

Builder method	When it is not called
`device`	`device::auto()`: Metal, then CUDA, then CPU, among the backends compiled in
`dtype`	the model's default: F32, except BF16 for `Qwen/Qwen3-Embedding-4B` and `-8B`, which turns into F32 on the CPU
`max_length`	the model's default length; any value is capped at its position limit
`batch_size`	32 texts per forward pass

An explicit BF16 request on the CPU is refused, because candle has no CPU BF16 matmul. Devices and dtypes has each dtype's limits, and testing the tolerance each dtype is held to.

Templates and token counts

candding/examples/query_passage.rs

//! bge prepends an instruction to queries and nothing to passages; `embed` applies no template.

use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("BAAI/bge-small-en-v1.5").build()?;
  let text = "what is a matryoshka embedding";
  let raw = model.embed(&[text], None)?;
  let query = model.query_embed(&[text], None)?;
  let cosine: f32 = raw[0].iter().zip(&query[0]).map(|(a, b)| a * b).sum();
  println!("query template: {:?}", model.descriptor().query_template);
  println!(
    "tokens without the template: {:?}",
    model.token_count(&[text])?
  );
  println!("cosine(raw, query) = {cosine:.4}");
  Ok(())
}

token_count returns the tokens the model sees after truncation, special tokens included and the template left out.

Where the weights come from

The builder reads the Hugging Face cache, ~/.cache/huggingface/hub by default, or a local directory holding config.json, tokenizer.json and the weights.
The default download feature fetches a missing model from the Hub in Rust; with default-features = false, the builder reads only the cache and local directories.
A gated repository, such as google/embeddinggemma-300m, needs its licence accepted on Hugging Face and a token in HF_TOKEN.

Model files and cache has the layout and the errors.

candding is Apache-2.0 OR MIT, but each model's weights carry their own licence, listed in the catalog: jinaai/jina-colbert-v2, jinaai/jina-reranker-v2-base-multilingual and naver/splade-code-06B are non-commercial, and google/embeddinggemma-300m comes under Gemma's terms.

Models outside the registry

candding/examples/custom_descriptor.rs

//! Load a BERT checkpoint that is not in the registry by supplying its descriptor.

use candding::Family;
use candding::ModelDescriptor;
use candding::Pooling;
use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let mut descriptor = ModelDescriptor::new(
    "sentence-transformers/all-MiniLM-L12-v2",
    Family::Bert,
    384,
    Pooling::Mean,
  );
  descriptor.default_max_length = 128;
  let model = TextEmbedding::builder(descriptor.id.clone())
    .descriptor(descriptor)
    .build()?;
  let rows = model.embed(&["a sentence outside the registry"], None)?;
  println!(
    "dim {} norm {:.4}",
    rows[0].len(),
    rows[0].iter().map(|v| v * v).sum::<f32>().sqrt()
  );
  Ok(())
}

A repository whose config.json names a supported model_type loads with or without a descriptor. ModelDescriptor::new takes the id, the family, the dimension and the pooling; every other field is public and starts with unit-length rows and no template, so the example sets only the length.

Load	Pooling, Dense modules and default length	Unit-length rows
Without a descriptor	read from the repository's sentence-transformers files	yes, whatever its `modules.json` lists
With a descriptor	the descriptor's; match its pooling to the repository's `1_Pooling/config.json`	as the descriptor's `normalize` field says

TextEmbedding::builder refuses an id that only another catalog carries with CanddingError::NonDenseModel, which names the builder that loads it; a local directory is never refused this way.