Testing

An embedding library can be wrong in a way that looks right: the vectors are finite, unit length and ranked plausibly, and only a side-by-side comparison with the reference implementation shows that a prefix is missing or an activation is the wrong one. That is why every model in the catalog carries stored reference vectors, and why the tolerances are part of the design rather than a knob.

Three tiers

Only the first tier runs by default. The golden and parity tests are marked ignored unless their feature is on, so a plain cargo test in a fresh checkout stays fast and hermetic while the model suites remain ordinary tests rather than a separate harness.

Golden references

References are generated by scripts/gen_reference.py with sentence-transformers on CPU in F32. The generator takes the descriptor from candding describe <id> --json, so it uses the same max_seq_length and the same query and passage templates the Rust side applies, and it pins the tokenizer's normalizer to the published tokenizer.json before encoding anything.

Nine fixture inputs are shared by every model and every suite: a short sentence, a long text well above any model's limit, a multilingual line mixing Latin, Japanese and accented French, an empty string, a whitespace-only string, a snippet of Rust code, and three texts of different lengths that exercise a mixed batch.

Each model directory under candding/tests/fixtures/ holds two files. reference.safetensors carries three tensors, raw, query and passage, one row per fixture input. meta.json records the model id and revision, the dimension, pooling, normalization flag and maximum length, the library versions the reference was produced with, and the Spearman correlation of the reference vectors on a small STS-B subset.

The ten tests per model

• raw embeds the fixture inputs with no template and compares every row with the raw tensor.
• query does the same through query_embed against the query tensor, so a missing or wrong prefix fails here.
• passage does the same through passage_embed against the passage tensor.
• single equals batched embeds each input alone and compares it with its row from a batched run, which is what catches padding leaking into a result.
• unit norm checks that every row of a normalizing model has length one.
• determinism embeds the same inputs twice and requires bit-identical output.
• truncation asserts that the long input comes back at exactly the model's maximum length and that the empty string still counts two tokens, the opening and closing special tokens.
• Matryoshka rejection asserts that a dimension request on a model without Matryoshka support is refused before any tokenization happens.
• STS smoke embeds the STS-B pairs, ranks the cosine similarities against the gold scores, and requires the Spearman correlation to stay within a fixed slack of the reference.
• snapshot head stores the first eight values of the first vector and compares against them on CPU, so an unintended change shows up as a readable diff.

Tolerances

Tolerances are never loosened to make a test pass. A cosine of 0.99 against a reference is a bug in the port, not floating-point noise, and the two columns exist because reduced precision and a different kernel genuinely move the last digits, not because a model is allowed to be a little wrong on a GPU.

Parity

Phase 0, the catalog's first family, is the whole registry today, and the parity test embeds the nine fixture inputs for every model in it, once on CPU and once on the compiled accelerator, with a required cosine of at least 0.999 for each vector. It is the check that keeps one code path honest: the model code has no backend branch, so a divergence can only come from the attention kernel or the dtype. On the Metal run every model cleared it with a minimum of 1.000000 at six decimals. CUDA has not been run, which is why the catalog shows untested in its CUDA column.

Two reference pitfalls

Both come from the Python side, and both are handled in the generator rather than in the tolerances.

• Several catalog repositories declare torch_dtype: float16 in config.json, and sentence-transformers would otherwise run the whole forward pass in fp16 and store an fp16-quality reference. The generator forces F32 parameters and asserts the parameter dtype before encoding.
• sentence-transformers rebuilds the tokenizer with a Lowercase normalizer when sentence_bert_config.json says do_lower_case: true, even where the published tokenizer.json says lowercasing is off. candding replays tokenizer.json as published, like FlagEmbedding, fastembed and text-embeddings-inference, so the generator pins the runtime normalizer to that file and prints whether the runtime differed.

Running it

just test                                   # unit tests, no weights
just fetch BAAI/bge-small-en-v1.5           # weights for one model
just test-models                            # golden tests on CPU
just test-metal                             # the same suite on the accelerator
just parity                                 # CPU against the accelerator
just golden BAAI/bge-small-en-v1.5          # regenerate one model's reference

To run one model's golden tests, filter by its module name: cargo test -p candding --features model-tests --test golden_bert bge_small_en_v1_5. Regenerate a reference only when the descriptor changes, and commit the new fixture with the change that required it.

CI mirrors that split. The ci workflow lints, runs the whole unit suite and runs the golden tests for two small models on every push to the default branch and every pull request, with the Hugging Face cache restored between runs so the weights are downloaded once. The metal workflow runs the same two models on an Apple runner, on demand or on a release tag, because a GPU runner is not something to spend on every commit.