Getting started

Add candding to a Cargo project, embed your first texts, and pick the device and dtype you want.

Install

Until the crate is on crates.io, depend on the repository:

[dependencies]
candding = { git = "https://github.com/rust-dd/candding" }

Enable a GPU backend with a feature flag: features = ["metal"] on Apple Silicon, features = ["cuda"] on NVIDIA. The default feature hf-cli lets the builder download a missing model through the hf command-line tool; install it with pip install huggingface_hub or uv tool install huggingface_hub.

Your first embedding

candding/examples/quickstart.rs

//! Embed two passages and a query on the auto-selected device and score them.

use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("BAAI/bge-small-en-v1.5").build()?;
  let passages = model.passage_embed(
    &[
      "Embeddings map text to vectors.",
      "Rust is a systems programming language.",
    ],
    None,
  )?;
  let query = model.query_embed(&["what do embeddings do"], None)?;
  for (i, passage) in passages.iter().enumerate() {
    let score: f32 = query[0].iter().zip(passage).map(|(q, p)| q * p).sum();
    println!("passage {i}: {score:.4}");
  }
  Ok(())
}

passage_embed and query_embed apply the model's templates; embed does not. For bge the query template prepends "Represent this sentence for searching relevant passages: " and the passage template is empty, so the two calls differ only for queries. Vectors are Vec<f32> of length model.dim(), L2-normalized when the descriptor says so, which is true for every Phase 0 model, so a dot product is a cosine similarity.

Choosing the device

candding/examples/devices.rs

//! Pick the device, dtype, max length and batch size explicitly.

use candding::DType;
use candding::TextEmbedding;
use candding::device;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("sentence-transformers/all-MiniLM-L6-v2")
    .device(device::cpu())
    .dtype(DType::F32)
    .max_length(256)
    .batch_size(16)
    .build()?;
  let rows = model.embed(&["hello world"], None)?;
  println!(
    "device {} dim {} max_length {} first {:.4}",
    device::describe(model.device()),
    model.dim(),
    model.max_length(),
    rows[0][0]
  );
  Ok(())
}

The builder picks Metal, then CUDA, then CPU depending on the features you compiled with; device::cpu(), device::metal(0) and device::cuda(0) override that. The default dtype is F32 on every backend. F16 and BF16 are opt-in and change results within the relaxed tolerances described on the testing page.

Templates and token counts

candding/examples/query_passage.rs

//! bge prepends an instruction to queries and nothing to passages; `embed` applies no template.

use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let model = TextEmbedding::builder("BAAI/bge-small-en-v1.5").build()?;
  let text = "what is a matryoshka embedding";
  let raw = model.embed(&[text], None)?;
  let query = model.query_embed(&[text], None)?;
  let cosine: f32 = raw[0].iter().zip(&query[0]).map(|(a, b)| a * b).sum();
  println!("query template: {:?}", model.descriptor().query_template);
  println!(
    "tokens without the template: {:?}",
    model.token_count(&[text])?
  );
  println!("cosine(raw, query) = {cosine:.4}");
  Ok(())
}

token_count reports the tokens the model will see after truncation, special tokens included, without a template.

Where the weights come from

The builder looks for the model in the Hugging Face cache (~/.cache/huggingface/hub by default). When it is missing and the hf-cli feature is on, it runs hf download with include patterns that skip ONNX and other exports. You can also pass a local directory that holds config.json, tokenizer.json and safetensors weights. The model files page has the layout and the gated-model notes.

Models outside the registry

candding/examples/custom_descriptor.rs

//! Load a BERT checkpoint that is not in the registry by supplying its descriptor.

use candding::BackendSupport;
use candding::Family;
use candding::ModelDescriptor;
use candding::Pooling;
use candding::TextEmbedding;

fn main() -> candding::Result<()> {
  let descriptor = ModelDescriptor {
    id: "sentence-transformers/all-MiniLM-L12-v2".to_string(),
    family: Family::Bert,
    dim: 384,
    default_max_length: 128,
    pooling: Pooling::Mean,
    normalize: true,
    query_template: None,
    passage_template: None,
    default_instruction: None,
    dense_dirs: Vec::new(),
    mrl: false,
    license: "Apache-2.0".to_string(),
    size_gb: 0.13,
    gated: false,
    phase: 0,
    backends: BackendSupport::untested(),
  };
  let model = TextEmbedding::builder(descriptor.id.clone())
    .descriptor(descriptor)
    .build()?;
  let rows = model.embed(&["a sentence outside the registry"], None)?;
  println!(
    "dim {} norm {:.4}",
    rows[0].len(),
    rows[0].iter().map(|v| v * v).sum::<f32>().sqrt()
  );
  Ok(())
}

Any repository whose config.json has a supported model_type loads with a descriptor you supply. Pooling and normalization come from the descriptor, so match them to the repository's 1_Pooling/config.json and modules.json.