JinaBERT

JinaBERT is BERT's attention names and post-LayerNorm residuals with the position table removed and a computed per-head bias put in its place: ALiBi, the only positional signal any of the six jina-embeddings-v2 checkpoints carry. Three block shapes share this encoder, told apart by their safetensors header rather than by config.json; the four that carry the base shape land in the catalog here, and the two that add query and key normalization wait for their own task.

What it is

An encoder-only transformer with bidirectional attention, a two-row token type table, post-LayerNorm residuals at an epsilon of 1e-12, a gated feed-forward, and no position table at all — position_embedding_type is "alibi", and the reference builds a position table only when it is not.

default_max_length is 8192 for all four, the value every one of their own sentence_bert_config.json files publishes; the sequence length and memory section below is why that number matters more here than anywhere else in the catalog.

ALiBi

Every attention score for head h between query position i and key position j picks up -slope_h * |i - j|: a fixed per-head penalty that grows with distance and depends on nothing either token contains.

The slope depends only on the head count. For a power of two n it is the closed form 2^(-8(i+1)/n); small-en's eight heads take this branch and range from 2^-1 down to 2^-8. For a head count that is not a power of two — the other three models here, all with twelve heads — the reference recurses one level: c is the largest power of two at or below n, the first c heads take that branch's own slopes, and the rest take every second entry of the 2c-head sequence, 2^(-4(2k+1)/c).

The sequence is not monotonic in head order: heads 8 through 11 carry four of the five largest slopes and sit last, so a sort() or a flip is a silent error rather than a no-op.

The bias is symmetric rather than causal-shaped — |i - j| does not care which side is larger — and it is never positive, zero only on the diagonal. It is also the model's only positional signal: with it absent, a reversed input would produce an exactly reversed output, because nothing else in the encoder distinguishes position i from position n - 1 - i.

Where the bias is added

ALiBi is added at the same seam the MPNet page describes for its relative position bias: after the query-key product is scaled, at the point the padding floor is added, so the two travel inside one additive tensor and layers::attention needs no branch for this family.

The composition is a selection rather than an addition. A padded key column is set to exactly the dtype's most negative finite value no matter what the bias there would have been, and a real key column keeps exactly the bias; nothing is summed. That distinction is not academic here: at 8192 tokens, the largest slope in a twelve-head model reaches a bias of about -5792 at the furthest pair, and composing that additively with F16's floor of -65504 would overflow the format's finite range to -inf, turning a fully padded row into NaN after the softmax. Selection keeps the padded column at exactly the floor regardless.

Sequence length and memory

This is the most practically useful fact on this page. The additive term ALiBi rides in is (1, heads, seq, seq) in F32: quadratic in the sequence length, linear in the head count, and, once anything in the batch is padded, linear in the batch size too.

An unpadded batch — every row the same length — keeps the tensor at its leading 1, so the Bias and Batch 1 columns are the whole cost no matter how large the batch is; the moment any row is shorter than its neighbours the tensor broadcasts to one copy per row, and the Batch 8 and Batch 32 columns are what that costs instead. Length-sorted batching is the difference between the 3.2 GB a single sequence at 8192 tokens costs and the 103 GB the same 32 of them cost unsorted.

The rule that follows: past roughly 1024 tokens, pair .max_length(n) with .batch_size(1) or another small batch, unless the caller already sorts inputs by length before batching them.

Encoder::max_position_embeddings() returns 8192 for all four because that is the length every repository's own config.json publishes, not because a table is being indexed at that height — ALiBi indexes nothing at all, so a request longer than 8192 would not go out of bounds. It would only leave the range the checkpoint was trained at, and cost more of the memory above.

candding builds this tensor once per forward and hands the same value to every layer; the reference computes attention_mask + bias inside each attention block instead, so at twelve layers it allocates the equivalent tensor twelve times over for one forward pass.

The gated feed-forward

The reference's feed-forward is a single fused (2 * intermediate, hidden) projection. candding splits it at load into the gate and the up-projection GatedMlp expects, which is exact rather than approximate, because the fused matmul's output columns are just the two halves side by side.

The activation comes from feed_forward_type, not from hidden_act: the reference's constructor reads this field to choose between nn.GELU() and nn.ReLU(), and hidden_act only feeds the non-gated feed-forward none of these four checkpoints instantiates. candding refuses anything but geglu at load rather than reading hidden_act and agreeing with the reference by coincidence.

For the base shape, the residual and the norm live inside this block rather than in the layer: JinaBertLayer::forward hands the attention output straight to the feed-forward with no add of its own, because the feed-forward already closes LayerNorm(gated(x) + x).

Every checkpoint says it is a BERT

All six repositories declare "model_type": "bert" in their config.json, and this is the thing a reader of this page is most likely to get wrong. candding resolves them to Family::JinaBert on a second field instead, "position_embedding_type": "alibi".

The two fields answer different questions: model_type names the training recipe transformers used, and position_embedding_type decides whether a position table exists in the checkpoint at all. Resolving these six on model_type alone would build a BertModel, which reads an embeddings.position_embeddings.weight tensor that none of the six safetensors files publish, so the load would fail outright — and before that failure was ever hit, the model table, models.json and this family's own page would already have named the wrong family.

Normalization

jinaai/jina-embeddings-v2-small-en publishes a modules.json of [Transformer, Pooling], with no Normalize module; candding's descriptor still says normalize: true for it, as it does for every catalog entry. The behavior is correct end to end: gen_reference.py encodes with the same flag the descriptor carries, so the reference candding is measured against is normalized too, and the visible consequence is that candding returns unit vectors where a bare SentenceTransformer.encode() of this repository would not. Every cosine between the two agrees; only the magnitudes differ.

The tokenizers

Two tokenizer families sit behind these four models. small-en and base-en share a 30528-entry WordPiece; base-de and base-zh each carry their own 61056-entry byte-level BPE with RoBERTa-style special tokens. token_count("") is 2 for all four — [CLS]/[SEP] for the WordPiece pair, <s>/</s> for the BPE pair — because every input is wrapped in the pair before it is counted.

base-zh looks like it should diverge and does not, which is worth more than an entry on a list. Its tokenizer_config.json names RobertaTokenizer and it publishes vocab.json and merges.txt beside tokenizer.json, which is exactly the shape a slow-tokenizer rebuild would be assembled from, and such a rebuild would carry no normalizer and a ByteLevel pre-tokenizer where the published file declares Sequence[NFC, Lowercase] and Whitespace — a difference that would change token ids wholesale rather than shifting their offsets. sentence-transformers loads RobertaTokenizerFast from the published file instead, so both pin stages match it and its Chinese text tokenizes the same either way. The catalog's pin-divergent models remain the three named on the reference-generation page. Predicting a divergence from a repository's file list is not the same as measuring one.

What the two byte-level BPEs do change is whitespace. base-de's keeps a whitespace-only input as six tokens where every other tokenizer in the catalog discards it, and the long fixture comes to 1022 tokens for base-de and 998 for base-zh against 1010 for the English pair.

Why the references come from a second environment

All six repositories ship their architecture as remote code, and that code predates transformers 5: under the environment the rest of the catalog's golden references are generated in, the class these checkpoints name fails to import. sentence-transformers does not raise when that happens. It silently builds the stock architecture config.json names instead, with every tensor the checkpoint does not supply randomly initialized, and encodes finite vectors from it without erroring, so a reference generated that way would be noise with nothing to say so.

These six references are generated through a second uv project instead, pinned to older sentence-transformers and transformers releases the repositories' own code does import cleanly under. gen_reference.py asserts the loaded class immediately after construction either way, so a repository whose remote code silently falls back to a stock architecture fails the generator loudly instead of writing a fixture that looks like a real one.

Pitfalls

Every figure below was measured by making the mistake on purpose in the committed source and embedding the nine reference fixtures with the real weights, on Metal in F32. The column is the smallest cosine any fixture reached against the sentence-transformers reference; the correct code reaches 1.0000 on both models to four decimals.

Two things in that table are worth more than the numbers.

The three slope-branch mistakes cost small-en exactly nothing, and for two different reasons. The forced branch and the descending sort are no-ops at any power-of-two head count, because that branch is what a power of two takes anyway and its slopes already descend; only a head count that is not a power of two can catch either. The naive sequence is narrower still: 2^(-8/n) equals 2^-1 only when n is exactly 8, so a four-head or sixteen-head model would catch it, and the unit test that sweeps n over 1, 2, 4, 8 and 16 does catch it with no checkpoint at all. What the three share is that small-en is blind to them, so a family verified on it alone would ship all three. A reversal is the opposite case: it reorders any sequence of more than one element, and it costs small-en more than any other single mistake in the table.

Nothing here throws. Every one of these produces finite, unit-length vectors that a smoke test accepts, and the mildest of them sits inside the range a reader might attribute to precision drift while being an arithmetic error. That is the argument for golden vectors over invariants for this family.

Two more pitfalls are about the checkpoints rather than the code.

F16 on CPU returns NaN for base-de and base-zh, for a six-token input as readily as for a long one, while F32 on the same device and F16 on Metal are both exact against the reference. candle's CPU f16 matmul accumulates in f16 where the Metal path does not, and these two checkpoints carry weights up to 20.4 and 30.2 where base-en peaks at 4.8 and small-en at 2.3. Use F32 on CPU for the multilingual pair, or F16 on an accelerator. The registry marks f16 supported on the strength of the accelerator runs, so nothing refuses the CPU combination today.

base-de is the catalog's only model whose tokenizer keeps a whitespace-only input. Its byte-level BPE makes six tokens of " \n\t " where every other published tokenizer here makes none. That is why it, alone, caught a reference generator that was silently stripping its inputs.

References