Testing

An embedding can be wrong and still look right: finite, unit length and plausibly ranked. Every catalog model is therefore compared with stored outputs of its reference implementation, at fixed tolerances.

What "verified" means

A catalog cell reads verified when that device's check below passed in that dtype, pending when a run is planned, untested when no run has reached the registry, and unsupported when the builder refuses the combination.

A dtype refused on every device, such as EmbeddingGemma's F16, reads unsupported on CUDA too. The unit tests, cargo test --workspace, run offline on synthetic checkpoints of all four pipelines; the golden and parity tests need the model-tests feature and the weights in the cache.

Golden references

scripts/gen_reference.py writes the dense references with sentence-transformers, in F32 on the CPU, forcing F32 parameters where a repository declares torch_dtype: float16. It reads candding describe <id> --json, so templates and maximum length match candding's, and pins the tokenizer's normalizer and pre-tokenizer to the published tokenizer.json. Other outputs are checked against their own reference:

• Inputs. Nine, shared by every suite: an empty and a whitespace-only string, a multilingual line, a Rust snippet, a text longer than 512 tokens, and three texts of different lengths for a mixed batch.
• Fixtures. A dense model's directory under candding/tests/fixtures/ holds reference.safetensors, a raw, query and passage row per input, and meta.json, which records the weight commit, the descriptor, the library versions and the token counts.
• Whitespace. The sentence-transformers release in scripts/remote-code/ strips each input and the one in scripts/ does not; the generator removes the strip, so both projects' references match candding, which never strips.
• Qwen3-Embedding-8B. Its reference is BF16 from Apple's MPS backend, since its F32 weights do not fit in the generating machine's memory.

Remote code

The jina-embeddings-v2, NomicBERT and GTE new repositories and two rerankers ship their architecture as remote code, so their references run in scripts/remote-code/, a second uv project that pins transformers 4. The generator runs remote code only when --trust-remote-code names the architecture at a pinned revision, and asserts that the loaded class is the repository's own, since a stock class of the same name initializes missing tensors randomly.

The dense tests

Each extra dtype has a module of six: raw, query, passage, single equals batched, unit norm and STS. A dtype the catalog refuses on the device becomes a check of the builder's error, and every module that computes reads back weights_dtype, so a load that dropped the requested dtype cannot pass in F32.

Tolerances

A BF16 model is held to the BF16 column; every other half-precision run, an F16 model or an F32 model against a BF16 reference, to the F16 column. The BF16 floor is lower because sentence-transformers' own BF16 runs miss the F16 floor. Tolerances are never loosened to let a result pass.

Sparse and multi-vector suites

Qdrant/bm25 has no golden module; a pinned unit-test suite checks it in the default run.

The rerank suite

golden_rerank has a module per checkpoint, such as ms_marco or qwen3_reranker_0_6b, with _f16 and _bf16 siblings. just golden-rerank <id> writes each fixture in F32 on the CPU from the descriptor in scripts/rerank/:

• a pair through AutoModelForSequenceClassification, on ids from the raw tokenizers backend, since a transformers tokenizer call turns a pair with an empty document into one segment;
• a prompt through AutoModelForCausalLM on the prompt rendered with the default instruction;
• the windowed reranker through its own rerank call, its windows recorded id for id.

Each module scores all 81 (query, document) pairs of the nine inputs, which candding batches per query with padding while the reference scores them one at a time. The default-dtype module also checks token counts, determinism, one pair's ids and that sorting a row is a total order. Before any score is compared, the fixture's head, activation and input must equal the registry entry's.

The 4B and 8B references load through an offloading route that just golden-rerank-route-check confirms bitwise equal to a plain load. The rerank page has each checkpoint's measured figures.

Running the suites

just test                     # unit tests, no weights
just fetch BAAI/bge-small-en-v1.5
just test-models              # the golden suites on the CPU
just test-metal               # the same on the accelerator
just parity                   # the CPU against the accelerator
cargo test -p candding --features model-tests --test golden_bert bge_small_en_v1_5
CANDDING_TEST_DEVICE=auto cargo test -p candding --features metal,model-tests --test golden_rerank ms_marco_f16:: -- --test-threads=1 --nocapture
just golden BAAI/bge-small-en-v1.5

There is one golden_<family> binary per dense family, plus golden_qwen3_4b, golden_qwen3_8b, golden_sparse, golden_multi_vector and golden_rerank; a module name filters one model. just test-models and just test-metal run the sparse, multi-vector and rerank modules one per process, since a module keeps its model loaded; the recipes include the large Qwen3 runs. Regenerate a reference only when the descriptor changes, and commit the fixture with that change.

CI

The ci workflow runs the unit suite and these golden modules on every push to main and every pull request; the rest of the catalog runs locally.

• The cuda workflow runs the same modules on an NVIDIA T4, or, dispatched with sweep: full, the golden modules of every entry but the non-commercial ones, as candding/tests/cuda_sweep derives them from the golden sources and the registry; it records BF16 as skipped below sm_80 and a module whose weights the T4 cannot hold as skipped, and contributing lists every workflow.
• EmbeddingGemma stays out of ci, since its repository is gated; the full sweep fetches it with the HF_TOKEN secret, which only its model-fetch step receives.
• naver/splade-code-06B, jinaai/jina-colbert-v2 and jinaai/jina-reranker-v2-base-multilingual stay out for their non-commercial licences: workflow_license_exclusion.rs fails the default test run if a workflow names one outside a comment.