Devices and dtypes

How candding picks CPU, Metal or CUDA, which dtypes each device runs, where the only backend branch lives, and the backend features.

A model runs on one device, the CPU, an Apple GPU through Metal or an NVIDIA GPU through CUDA, in one dtype, F32, F16 or BF16. The model code is the same on every device: the device only decides which attention kernel runs, and the dtype decides the weights' memory and precision.

Selection order

A builder with no .device(...) calls device::auto(). A backend is available only when the crate is compiled with its feature, so a default build always lands on the CPU.

Function	Device
`device::auto()`	Metal, then CUDA, then the CPU, among the compiled-in backends, at ordinal zero
`device::cpu()`	the CPU
`device::metal(ordinal)`, `device::cuda(ordinal)`	that GPU, with no fallback: `CanddingError::BackendNotCompiled` names the feature a build lacks, and `CanddingError::Device` carries candle's message
`device::describe(&device)`	`cpu`, `metal` or `cuda`, the name log lines and errors use

Forward passes on Metal are serialized across the process, whatever the model type, to avoid a buffer-reuse hazard in candle's Metal allocator. On the CPU, concurrent calls on one instance run concurrently.

Dtypes

Each registry entry has a default dtype, which .dtype(...) overrides. The weights are cast as they load, so a BF16 checkpoint loaded as F32 is upcast losslessly and one loaded as F16 is rounded; the returned vectors are always f32.

Models	Default dtype
Qwen/Qwen3-Embedding-4B and -8B, Qwen/Qwen3-Reranker-4B and -8B	BF16, their checkpoints' own dtype, since their F32 weights alone take 16 GB or more
every other model	F32

Dtype	CPU	Metal	CUDA
F32	runs	runs	verified on an NVIDIA T4 for every entry the testing table does not list
F16	runs, with candle accumulating each matmul in F16; an entry whose results miss its tolerance there is refused	runs	as F32; refused for the entries that table lists as `unsupported`, and where the registry refuses F16 on every device
BF16	refused, since candle has no CPU BF16 matmul; a BF16 default falls back to F32 with a warning	runs	untested, since candle builds BF16 kernels only for `sm_80` and newer and the T4 is `sm_75`, except where the registry refuses BF16 on every device

A refused combination fails at build time: CanddingError::UnsupportedOnDevice for a dtype the entry cannot run on that device, and CanddingError::UnsupportedDtype for one refused on every device, such as EmbeddingGemma's F16, whose activations overflow it. Builders refuse from the rows the catalog shows, so the catalog lists what loads where; the family pages give each refusal's cause, and testing defines verified.

LayerNorm and RMSNorm compute in F32 and return the input dtype, which keeps half-precision runs within their tolerance.

Loads that cannot fit

A builder sizes the weights before it reads any of them: from the safetensors headers, from a torch pickle's or NumPy file's size scaled to the dtype it loads in, or, for a model not downloaded yet, from the catalog's download size before the download starts. When they need more bytes than the device's budget, the build fails with CanddingError::InsufficientMemory, whose message gives both sizes in GB and the ways out: a smaller dtype, another device or a smaller model. Activations, which grow with the batch size and the length, come on top of the weights and are not counted.

Device	Budget
CPU	the machine's physical memory, `MemTotal` on Linux and `hw.memsize` on macOS: past the free pages the system compresses and swaps, which is slow but correct, so only a model larger than the machine is refused
Metal	the GPU's recommended working set, less what the process already holds on it
CUDA	the free memory the driver reports

One attention branch

Model code has no #[cfg] and no device checks: the shared attention function picks candle's fused SDPA on Metal at the head sizes it supports, and two matmuls around a masked softmax everywhere else, without changing the math. The additive mask uses the dtype's most negative finite value, so an all-padding row stays finite; flash attention on CUDA is on the roadmap.

Memory at long lengths

The matmul path materializes a (batch, heads, seq, seq) scores tensor in every layer, quadratic in the length. MPNet's relative bias and JinaBERT's ALiBi also build an additive mask of that shape, (1, heads, seq, seq) while no row is padded; Metal's fused kernel skips the scores but takes the mask expanded to every head and row.

Case	Size
`BAAI/bge-m3`, one input at its default 8192 tokens, matmul path	about 4 GB of scores per layer
`sentence-transformers/all-mpnet-base-v2`, batch 32 at 384 tokens, padded	about 226 MB of F32 mask, the size of the scores tensor beside it

For long inputs, pair .max_length(n) with .batch_size(1) or another small batch; the JinaBERT page has the ALiBi mask's size per length and batch.

Backend features

Feature	Backend	Notes
none	CPU	always compiled in
`metal`	Apple GPU	`device::auto()` prefers it
`cuda`	NVIDIA GPU	needs the CUDA toolkit
`cudnn`	NVIDIA GPU	implies `cuda` and adds cuDNN kernels; no workflow builds it
`accelerate`	CPU	Apple's BLAS
`mkl`	CPU	Intel's BLAS

Every model family compiles in unconditionally; the API page lists the remaining features, download and model-tests.