NomicBERT

NomicBERT replaces BERT's position table with rotary embeddings and its feed-forward with a SwiGLU block, and fuses the query, key and value projections into one matrix. Three checkpoints share it: nomic-ai/nomic-embed-text-v1, nomic-ai/nomic-embed-text-v1.5 and Snowflake/snowflake-arctic-embed-m-long, which declares the same model_type.

What it is

An encoder-only transformer with bidirectional attention, a two-row token type table and post-LayerNorm blocks at 1e-12 with no final norm. The rotation spans the full head, rotate-half, at the base of 1000 every checkpoint declares, and the bias-free feed-forward is fc2(fc11(x) * silu(fc12(x))). The config must say activation_function: "swiglu" with prenorm and causal both false.

How candding implements it

The three checkpoints differ in how the rotary table behaves past the trained length of 2048 tokens:

base' = base * ((factor * seq / max_trained) - (factor - 1)) ^ (dim / (dim - 2))

The rescaled table is built for the longest sequence in the current batch, so a text in a batch that passes the trained length embeds differently from the same text alone. No golden fixture reaches 2048 tokens, so no test compares rescaled vectors with the reference; unit tests check the formula itself.

nomic-ai/nomic-embed-text-v1.5's model card applies a LayerNorm with no affine parameters, at 1e-5, between mean pooling and Matryoshka truncation, a step its modules.json omits; its descriptor sets layer_norm_after_pooling, which the pipeline runs for it alone.

The default length is 8192 for all three, from sentence_bert_config.json. Rotary tables grow to whatever a batch needs, so the family reports no position limit, and the default or .max_length(n) caps an input.

Pooling and templates

The nomic model card also names clustering: and classification: prefixes, which a descriptor has no slot for: prepend one to the text and call embed. nomic-ai/nomic-embed-text-v1.5 supports Matryoshka truncation and ships no Normalize module; the pooling and templates page has both. All three use the bert-base-uncased WordPiece tokenizer.

Supported models

Pitfalls

• rotary_emb_base has no safe default: read it from the checkpoint. A wrong base still gives finite, unit-length vectors.
• Skipping the dynamic NTK rescaling for v1 or the Snowflake checkpoint, or applying it to v1.5, changes nothing at or under 2048 tokens, so only a longer input shows it.
• fc12 is the gate and fc11 the up-projection; the two have the same shape, so a swap still runs.
• layer_norm_after_pooling belongs to v1.5 alone; set wrongly, it changes each vector's direction and nothing else.
• Wqkv's rows are (three, heads, head_dim); read as (heads, three, head_dim), every projection mixes query, key and value in a tensor of the right shape.
• config.json's max_position_embeddings is not the length cap; used as one, it caps v1.5 at 2048 tokens.

References