Sparse output

A sparse embedding is a row of (index, positive weight) pairs over a model's own output space instead of a fixed-width dense vector, ascending by index and deduplicated by keeping the maximum weight when two occurrences share one.

pub struct SparseEmbedding {
  pub indices: Vec<u32>,
  pub values: Vec<f32>,
}

SparseTextEmbedding::builder(id).build()?.embed(texts, None)? returns Vec<SparseEmbedding>, one row per input text, the sparse counterpart of TextEmbedding::embed's Vec<Vec<f32>>.

The index

What an index names is head-specific, not a candding convention, and both of today's heads name it the same way: the model's own token vocabulary id, not a hash or a learned code. SpladeMlm (prithivida/Splade_PP_en_v1) indexes the model's own English WordPiece vocabulary; M3Linear (BAAI/bge-m3, a second catalog entry beside its dense one) indexes the XLM-RoBERTa SentencePiece vocabulary the dense entry already uses for input ids. A future hashed head, such as the BM25-style entries on the roadmap, would use the same SparseEmbedding type with indices that name a hash bucket instead — nothing in the type itself assumes a vocabulary.

The two heads

SpladeMlm reuses the BERT encoder and adds the standard masked-language-model head, pooled by an unconditional relu then log1p on every logit and a maximum over the sequence; the tied embedding table doubles as its decoder weight, so nothing about the head is a separately trained matrix. M3Linear reuses the XLM-RoBERTa encoder and adds a single Linear(hidden, 1) per position, relu'd and scattered into its output column by that position's own input id, keeping the larger of two logits when a token repeats and dropping the four special ids (<s>, <pad>, </s>, <unk>) the reference always drops. One of bge-m3's nine golden fixtures is genuinely empty on both sides, empty: it tokenizes to nothing but those four dropped ids, so nothing survives the scatter. SPLADE has no dropped-id list, so its own empty and whitespace rows still carry seven entries each, the masked-language head's own prediction for the CLS and SEP positions. bge-m3's whitespace row is empty on the reference side only, not candding's: the published tokenizer.json candding replays gives that input three ids, 0, 6, 2, where the plain AutoTokenizer the reference is generated with gives two, 0 and 2; the extra id survives the scatter as one entry, index 6, weight 0.027290672, that the reference never produces. That disagreement is a tokenizer fact rather than a numeric one, identical in F32, F16 and BF16, and the golden suite's sparse_meta.json records it as a required divergence rather than a tolerance the row is compared against — the two sides staying different is what makes the row pass.

The M3Linear row above is the reference's own counts; candding's own whitespace count is 1, not 0 (see above), so candding's own row totals 210, not 209.

A sibling catalog, not a mode

Sparse output is a second, independent object graph beside the dense one rather than a flag threaded through it. SparseModel is its own trait, forward(&EncoderInput) -> Result<Vec<SparseEmbedding>>, separate from Encoder, because a sparse head's forward pass already is the whole postprocessing step: there is no shared pooling or normalization stage outside it the way CLS, mean or last-token pooling sit outside every Encoder. SparseTextEmbedding and SparseTextEmbeddingBuilder parallel TextEmbedding and TextEmbeddingBuilder with their own methods rather than extra parameters on the dense ones, SparseDescriptor parallels ModelDescriptor with no dim, pooling, normalize or template fields because a sparse row's width is its head's own vocabulary size and the head decides what pooled and normalized mean, and registry::sparse is a second catalog beside registry's own, with its own all() and lookup().

The two catalogs are independent lists an id can sit in one, the other or both: BAAI/bge-m3 is a dense entry in registry and a second, separately statused entry in registry::sparse, one set of encoder weights behind two catalog rows. Registering a sparse entry never touches the dense list, so the dense catalog's own count is unaffected by what the sparse list carries: 35 models across nine families, exactly as before this phase. website/public/models.json and the catalog table read the dense list alone, by design rather than by omission — a sparse row has no dim, pooling or normalize to put in that table's columns.

Scoring two sparse embeddings

SparseEmbedding::dot is the scoring rule both references declare: the sum of the products over indices present on both sides, computed as a merge over the two ascending index lists rather than a search, and zero when either side is empty. A caller scores a query against a passage the same way a dense caller takes the dot product of two vectors, except neither side is normalized to unit length the way a pooled dense vector is, so a sparse score is not bounded to [-1, 1].

let model = candding::SparseTextEmbedding::builder("prithivida/Splade_PP_en_v1").build()?;
let query = model.embed(&["a quick fox"], None)?;
let passage = model.embed(&["The quick brown fox jumps over the lazy dog."], None)?;
let score = query[0].dot(&passage[0]);

Half precision

Both catalog entries record f16 as Unsupported, and SPLADE's bf16 is Unsupported too — deliberately conservative, not only a measurement of last resort. SparseEntry carries one status per dtype with no per-device split the way the dense registry's accel_only_dtypes gives Entry, so a dtype that clears on the accelerator and misses on CPU has no way to read Verified for the device where it clears and Unsupported for the one where it does not; adding a per-device status without a matching per-device tolerance column would only be half of that fix. bge-m3's f16 is the case that costs something: on Metal it clears every one of its eight non-whitespace fixtures comfortably, and it only misses on CPU, candle's already-documented CPU F16 matmul accumulation. SPLADE's f16 would stay Unsupported even with that split, because its own Metal sweep already misses two of the nine fixtures on its own terms. bf16 splits the other way: candle has no CPU BF16 matmul at all, so the dtype was always Metal-only regardless of any per-device field, and both entries' bf16 statuses are plain measurements rather than a conservative placeholder — bge-m3's clears and reads Verified, SPLADE's misses on multilingual and reads Unsupported.

The columns above were never loosened to make a status fit, and they are not a generous allowance either: they were calibrated in Python, never in candle. SPLADE's came from sentence-transformers' own SparseEncoder, a genuine half-precision run of the real reference library end to end; bge-m3's came from casting only its 1024-to-1 head while the encoder underneath stayed at F32, which never measured a half-precision encoder at all, candle's or PyTorch's. Neither describes candle's own processor-side half-precision arithmetic, which accumulates in half precision on CPU and is measurably worse: SPLADE's own CPU F16 run misses its first-checked fixture, short, at max-abs 3.30e-2 — roughly seven times the PyTorch reference's own worst F16 max-abs of 4.76e-3, the same accumulation the devices and dtypes page documents for the dense tolerance table, exposed here by columns calibrated an order of magnitude tighter. That gap is this backend's own pitfall, not a property of either model.

Once a dtype is Unsupported, its golden module stops computing anything to compare: sparse_dtype_runs_on turns it into a refusal check, so the module still runs and still passes, but it asserts only that SparseTextEmbeddingBuilder::build() returns CanddingError::UnsupportedDtype, never that a vector was measured. The suite is 32 of 32 on CPU and 32 of 32 on Metal, and not all 32 come from the six model modules: 10 of the 32 are harness unit tests over hand-built SparseEmbeddings and a Spearman helper, identical on both devices, which never load a model and pass with no weights on disk. Of the 22 that do load a model, the split is device-specific: on CPU, 6 compare a sparse vector against its reference, 4 more exercise a real forward pass, and 12 assert a refusal, across four modules that refuse there, not three — splade_f16, splade_bf16 and bge_m3_f16 because those dtypes are Unsupported, and bge_m3_bf16 too, because candle has no CPU BF16 matmul at all. On Metal the same four figures are 9, 4 and 9, across three refusing modules, because bge_m3_bf16 clears there and joins the ones that compute. Either way, a pass in a refusal module means the builder now refuses before computing anything, not that half precision was cleared again on this run.