Devices and dtypes

Selection order

A builder with no explicit device calls device::auto(), which tries Metal, then CUDA, then CPU. The first two are only available when the crate was compiled with the matching feature, so a default build always lands on CPU and a build with features = ["metal"] on an Apple machine always lands on the GPU. Ordinal zero is used for both accelerators.

device::cpu(), device::metal(ordinal) and device::cuda(ordinal) set the device explicitly, and a device that cannot be created returns CanddingError::Device with the message candle produced rather than falling back silently. device::describe turns a device into the short name that appears in log lines and errors: cpu, metal or cuda. The CLI exposes the same four choices as --device auto|cpu|metal|cuda, and auto is its default.

Dtypes

The builder loads weights in F32 on every backend unless you ask for something else, and the dtype you pass is the dtype the safetensors are memory-mapped into. F16 and BF16 are opt-in through .dtype(...), halve the memory the weights need, and shift the vectors enough that the test suite applies its relaxed tolerances to them.

Whatever the weights are in, the pooled vectors are upcast to F32 before Matryoshka truncation and before L2 normalization, and the rows handed back are always Vec<f32>. The one other place dtype matters is the additive attention mask: masked positions get the most negative finite value of the dtype, -65504 in F16 and the corresponding floor in BF16 and F32, rather than negative infinity, because a row that is entirely padding would otherwise come out as NaN. When no row in a chunk is padded there is no additive mask at all.

One attention branch

Model code contains no #[cfg] attributes and no device checks. The single branch is in layers::attention::scaled_dot_product, and it chooses a kernel rather than changing the math:

• On Metal, when the head size is one of 32, 64, 72, 80, 96, 128 or 256, the call goes to candle's SDPA kernel, with the mask pre-broadcast to (batch, heads, seq, seq), made contiguous and cast to the query dtype the way that kernel expects.
• Everywhere else, and for any other head size, scores are computed as a matmul, scaled, given the additive mask and softmaxed before the second matmul. This is the CPU path in every build, because candle's SDPA has no CPU implementation.

A head size of 512 is deliberately absent from that list: the Metal kernel rejects F32 at that size, so those models take the matmul path on every backend. Flash attention on CUDA is on the roadmap and would be a third kernel behind the same function.

Parity

Because the two paths are supposed to be the same computation, they are tested as such. The parity suite embeds the shared fixture inputs for every Phase 0 model twice, once on CPU and once on the compiled accelerator, and asserts a cosine of at least 0.999 for each vector. On the Metal run all thirteen models cleared it with a minimum of 1.000000 at six decimals. CUDA has not been run yet, which is why the catalog shows untested in its CUDA column instead of a promise; the testing page describes both suites and how to run them.

Feature flags

All model families compile in unconditionally: there is no per-model or per-family feature, only backends and the two switches above.