NomicBERT

NomicBERT keeps two of BERT's tensor names, embeddings.word_embeddings and embeddings.token_type_embeddings, and gives everything else its own: a learned position table replaced by rotary embeddings, a fused attn.Wqkv projection split into query, key and value at load, and a feed-forward of three separate tensors — mlp.fc11, mlp.fc12 and mlp.fc2 — rather than one fused matrix. Three checkpoints share it in the catalog: nomic-ai/nomic-embed-text-v1, its v1.5 successor, and Snowflake/snowflake-arctic-embed-m-long, a checkpoint published under a different repository that declares the same model_type and loads through the same code path. All three are Apache-2.0, ungated, 768-dimensional and 0.55 GB on disk.

What it is

An encoder-only transformer with bidirectional attention, a two-row token type table, post-LayerNorm residuals at an epsilon of 1e-12, and a SwiGLU-gated feed-forward: activation_function: "swiglu" is the only value this family's config accepts, and prenorm and causal must both be false. Its positional signal is rotary rather than a learned table — rotary_emb_fraction: 1.0 rotates the full head and rotary_emb_interleaved: false takes the rotate-half convention layers::rope::RotaryEmbedding already implements for Qwen3 and Gemma3 — and every one of the three checkpoints sets a base of 1000 for rotary_emb_base, a required field with no default. RoFormer's own paper put the base at 10000; the catalog's other two rotary families default higher still, Qwen3 and Gemma3's global layers to 1000000 and Gemma3's local sliding-window layers to 10000; this family's 1000 is the smallest base of any of them.

The shape above is identical across all three checkpoints, and so are layer_norm_epsilon at 1e-12 and rotary_emb_base at 1000; what differs between them is rotary_scaling_factor, max_trained_positions and max_position_embeddings, covered in Rotary at base 1000 and the length ruling below. default_max_length is 8192 for all three regardless, each repository's own sentence_bert_config.json value — independent of config.json's own max_position_embeddings, which is 8192 for v1, 2048 for v1.5, and absent entirely for the Snowflake checkpoint. The length ruling section below is why that difference does not become a limit.

The tensors

112 tensors, none of them carrying a bias: embeddings.word_embeddings.weight [30528, 768], embeddings.token_type_embeddings.weight [2, 768], emb_ln.{weight,bias} [768], and per layer encoder.layers.{i}.attn.Wqkv.weight [2304, 768], attn.out_proj.weight [768, 768], mlp.fc11.weight [3072, 768], mlp.fc12.weight [3072, 768], mlp.fc2.weight [768, 3072], and norm1.{weight,bias} and norm2.{weight,bias}, both [768]. No position table and no pooler.

Wqkv's 2304 rows are query, key and value stacked in that order: rows 0..768 are the query for every head, 768..1536 the key, 1536..2304 the value, which is also how the reference's own rearrange call reads them ("... (three h d) -> ... three h d", with three as the outermost axis). candding's split_qkv narrows those same three row ranges directly; reading the fused weight per head instead, as (heads, three, head_dim), would produce a tensor of the identical shape whose every projection mixes the three, with nothing about its shape or its norm to say so.

Each block is h = norm1(attn(h) + h), then h = norm2(mlp(h) + h), with no final norm after the last layer. Every LayerNorm in the encoder, emb_ln included, uses an epsilon of 1e-12, and attention scales by 1 / sqrt(64).

Rotary at base 1000

v1 and the Snowflake checkpoint rescale the rotary base past their trained length with the reference's dynamic-NTK formula, evaluated in f64 throughout:

base' = base * ((factor * seq / max_trained) - (factor - 1)) ^ (dim / (dim - 2))

with factor: 2, max_trained: 2048 — the shared trained length both checkpoints use, which v1's own config.json never sets and reads the reference constructor's own default instead — and dim: 64, the head size. At or below max_trained the base is unchanged.

The five figures below are computed directly from that formula, not measured against any reference:

v1.5 sets rotary_scaling_factor: null: its rotary table uses the plain configured base at every length, including past its own max_trained_positions of 2048, and the reference embeds it unscaled all the way to 8192 rather than applying the rescaling above.

Measured rather than computed, for the Snowflake checkpoint alone: concatenating the long fixture text with itself until the reference tokenizer counted 3026 tokens, then embedding it alone on CPU in F32 at max_length 8192, candding's dynamic-NTK branch reaches cosine 1.000000 and a max absolute difference of 3.36e-07 against a sentence-transformers reference encoded with the same rescaling; a second reference run of the same input with rotary_scaling_factor forced to null in config_kwargs reaches cosine 0.539593 and a max absolute difference of 0.521294 against candding's own output.

candding's cache follows the batch rather than the row: a fresh table past the trained length is sized to the longest sequence in the current batch, so the same text can pick up a different rotary table depending on what it is padded alongside once that length is exceeded. The reference's own cache only ever grows — it remembers the longest length any call has reached and never shrinks — where candding discards and rebuilds a fresh table on every call past the trained length instead of keeping one around. Every fixture the catalog measures stays at or under the trained length, where both conventions agree exactly, so no fixture depends on either divergence.

The feed-forward halves

The feed-forward is three separate, bias-free tensors rather than one fused matrix: fc11 and fc12 each project the 768-wide hidden state up to the 3072-wide intermediate, and fc2 projects the result back down. fc12 is the gate and fc11 the up-projection: y = fc11(x) * silu(fc12(x)), then fc2(y).

Getting the two backwards raises nowhere: both tensors are the same shape, so swapping which one is the gate still produces a finite result, just a different one. NomicBertMlp's own unit test pins the assignment down with a one-dimensional toy config rather than a checkpoint: gating fc12 correctly gives 2.857722, gating fc11 instead gives 2.193176, two distinguishable values from a config with no checkpoint involved at all.

LayerNorm after pooling

nomic-embed-text-v1.5's model card documents its own encode as four steps, with no affine parameters and an epsilon of 1e-5 on the LayerNorm:

x = mean_pooling(token_embeddings, attention_mask)
x = F.layer_norm(x, normalized_shape=(dim,))
x = x[:, :matryoshka_dim]
x = F.normalize(x)

Neither modules.json nor 1_Pooling's own config lists that LayerNorm at all.

candding's pipeline runs a matching step for every catalog model, gated by the descriptor's layer_norm_after_pooling flag, after the Dense modules and before Matryoshka truncation: postprocess::layer_norm centers each row on its own mean and divides by its own standard deviation at an epsilon of 1e-5, with no learned weight or bias — which is what "no affine parameters" means for a LayerNorm the checkpoint carries no tensors for at all. v1.5 is the only catalog entry with the flag set; v1, the Snowflake checkpoint, and every other family in the catalog leave it false and skip the step entirely.

The length ruling

default_max_length comes from sentence_bert_config.json, not from config.json's own max_position_embeddings — which is why it reads 8192 for every one of the three even though v1.5's own config caps at 2048 and Snowflake's does not set the field at all.

Nothing about any of those per-checkpoint figures is enforced by this family's code. NomicBertModel::max_position_embeddings() returns usize::MAX rather than any of them, because rotary tables are built to whatever length a batch needs rather than indexed into a table of fixed size, and a hard cap at v1.5's own 2048 would truncate exactly where its reference embeds unscaled instead. The pipeline's own cap — the smaller of the requested length and max_position_embeddings() — becomes a no-op for every registered NomicBERT entry as a result: the descriptor's default_max_length of 8192, or a caller's own .max_length(n), is what actually limits a request.

Templates

v1 and v1.5 share two prefixes from their model card: search_query: for a query and search_document: for a passage, each simply placed in front of the input with no other formatting. candding's descriptor carries them as query_template: Some("search_query: {text}") and passage_template: Some("search_document: {text}"), with template::apply's {text} placeholder standing in for the concatenation.

The same card documents two further task prefixes, clustering: and classification: , that ModelDescriptor has no slot for: it carries only a query and a passage template. A caller who needs one of those two prepends it to the text directly and calls the untemplated embed rather than query_embed or passage_embed.

Snowflake's model card and its config_sentence_transformers.json agree on a single query-only prompt, Represent this sentence for searching relevant passages: ; candding's descriptor reads it from the config file, carrying it as query_template with no passage_template at all — passage text for this checkpoint goes through unprefixed.

The tokenizers

All three read a bert-base-uncased WordPiece: lowercase, a 30522-entry vocabulary, [PAD] 0, [CLS] 101, [SEP] 102. The embedding table is six rows wider, [30528, 768], than the tokenizer can address — pad_vocab_size_multiple: 64 rounds 30522 up to the next multiple of 64, and those six extra rows are never reached. "" and a whitespace-only input both tokenize to [101, 102], [CLS] and [SEP] with nothing between them, because this WordPiece's own pre-tokenizer discards whitespace the way the catalog's other bert-base-uncased tokenizers do.

v1 and v1.5 publish byte-identical tokenizer.json files. Snowflake's differs only in its declared padding and truncation metadata, not in its vocabulary, so the same input tokenizes identically on all three.

Why the references come from the second environment

All three repositories ship their architecture as remote code: nomic-ai/nomic-bert-2048's own code revision provides it for v1 and v1.5, and Snowflake publishes its own copy under its own repository and its own code revision. That code's rearrange calls import einops, a dependency neither of candding's two reference-generation uv projects carried before this family.

Adding it to scripts/remote-code's own pyproject.toml — the one new dependency this family needed, Python rather than Rust — reuses the environment the six JinaBERT checkpoints already generate their references through: an older, pinned sentence-transformers and transformers release.

Running any of this under the wrong environment does not raise on its own. Without --trust-remote-code, refuse_untrusted_remote_code catches a repository whose config.json carries auto_map before a fixture can be written from it; loaded under --trust-remote-code but the wrong transformers release, the remote class can still fail to import, and sentence-transformers falls back silently to the stock architecture config.json names — building every tensor the checkpoint does not supply from a random initialization and encoding finite vectors from it with no error at all. assert_architecture is what catches that instead: it checks the loaded class's own name against the one the model is supposed to be, immediately after construction, so a checkpoint that silently fell back fails the generator loudly rather than writing a fixture indistinguishable from a real one.

A second, independent check runs v1 and v1.5 with no trust_remote_code at all: scripts/'s own transformers release ships a native transformers.models.nomic_bert.modeling_nomic_bert.NomicBertModel for this model_type, so AutoModel.from_pretrained loads both without touching either repository's own remote code. Encoding the nine fixture texts through that native class — tokenizing from the repository's own published tokenizer.json, mean-pooling with the attention mask, and reproducing raw's own post-processing by hand, v1.5 additionally through F.layer_norm before normalizing — measured cosine 1.000000 and max-abs 3.58e-07 for v1 and cosine 1.000000 and max-abs 1.12e-07 for v1.5 against the two fixtures' own raw matrices, at the fixture lengths: the longest fixture is 1010 tokens, under this family's shared 2048 trained-position length, so the comparison does not exercise v1's dynamic-NTK branch.

candding itself runs none of this code. hub::INCLUDE_PATTERNS downloads no .py file from any of the three repositories, and the two modelling files this family's config and encoder were read from exist on disk only as a specification, read by a person rather than executed by the library.

Supported models

Pitfalls

Every mistake below produces a finite, plausible-looking vector rather than an error or a NaN, which is why each needs a golden reference rather than a smoke test to catch.

rotary_emb_base given a fallback default instead of read from the checkpoint's own config.json. The field has no safe default to fall back to: the crate's other rotary families do not agree with each other either, 1000000 for Qwen3 and Gemma3's global layers, 10000 for Gemma3's local ones, and none of those matches this family's own 1000. The table is still a well-formed set of cosines and sines, every attention score is still finite, and the pooled, normalized output is still a unit vector — nothing anywhere raises, and every position past the first is simply rotated by the wrong angle.

The dynamic-NTK rescaling skipped for v1 or the Snowflake checkpoint. At or under the trained length of 2048 tokens this is exactly a no-op, so a short-input check shows nothing wrong at all; only a request past 2048 tokens is computed at the wrong base, silently.

The dynamic-NTK rescaling applied to v1.5 despite its null scaling factor. The same blindness in reverse: invisible at or under 2048 tokens, and past it, a finite table built at a base the checkpoint never uses.

fc11 and fc12 swapped, gating the up-projection instead of the gate. Both tensors are the same shape, so the swap produces a different, still-finite value with nothing about the shapes or the forward pass to distinguish a correct gate from a swapped one.

layer_norm_after_pooling left off for v1.5, or turned on for v1 or the Snowflake checkpoint. The pooled vector is finite either way, and after normalization it is unit length either way; only its direction changes, so a check that verifies shape and finiteness alone sees nothing wrong.

Wqkv read as (heads, three, head_dim) instead of (three, heads, head_dim). The result is a same-shaped tensor whose every head's query, key and value are each a mixture of the real ones — still finite, still attending to something, still a finite embedding at the end.

config.json's own max_position_embeddings used as the effective length ceiling instead of sentence_bert_config.json's default_max_length. For v1.5 this silently caps every request at 2048 instead of 8192: nothing raises, and no input already under the shorter limit looks any different.

References