JinaBERT

JinaBERT is BERT's attention names and post-LayerNorm residuals with the position table removed: a computed per-head bias, ALiBi, is the only positional signal. The six jina-embeddings-v2 checkpoints run on it, in three block shapes, as do the reranker jinaai/jina-reranker-v1-tiny-en and the encoder of the miniCOIL sparse head.

What it is

An encoder-only transformer with bidirectional attention, a two-row token type table, post-LayerNorm residuals at an epsilon of 1e-12, a geglu feed-forward, the only feed_forward_type candding accepts, and no position table: every attention score for head h between positions i and j picks up -slope_h * |i - j|. small-en has eight heads; every other checkpoint, the reranker included, has twelve.

How candding implements it

Every checkpoint declares "model_type": "bert", so candding resolves the family from "position_embedding_type": "alibi", and two marker tensors in layer 0 of the safetensors header decide the block shape:

The QK-norm shape adds a LayerNorm over the full query and key projections, before the head split. The split-norm shape adds the same norms and rearranges the layer around a feed-forward with no norm or residual of its own:

residual = layer_norm_1(x + attention(x))
output   = layer_norm_2(residual + mlp(residual))

A power-of-two head count n takes the slopes 2^(-8(i+1)/n); twelve heads take the first eight of the eight-head sequence, then every second entry of the sixteen-head one. jinaai/jina-reranker-v1-tiny-en's bundled code halves the slopes of heads 8 to 11, which its entry records as AlibiSlopeVariant::HalvedInterpolatedTail; every embedding checkpoint is Canonical.

Sequence length and memory

The additive term is (1, heads, seq, seq) in F32, quadratic in the length, and one copy per row once any row in a batch is padded. On the matmul path an unpadded batch costs the batch-1 column; on Metal the fused kernel takes the term expanded to the full batch.

Past about 1024 tokens, pair .max_length(n) with .batch_size(1) or another small batch. The default length is 8192 for all six embedding checkpoints, the value each repository publishes; ALiBi indexes no table, so 8192 marks the trained range rather than an array bound.

Pooling and templates

The six embedding checkpoints average the real tokens and have no templates, instruction placeholder or Matryoshka support, so embed, query_embed and passage_embed agree; small-en and base-code ship no Normalize module. small-en and base-en share a WordPiece tokenizer, and the other four each publish a byte-level BPE.

Supported models

The reranker scores a pair through its BertPooler and returns a probability; the rerank page has its status.

Pitfalls

• A block shape read wrong still loads and returns finite unit vectors; only a golden reference separates the three.
• The QK norms span the whole projection, not one head; moved after the head split, the weight no longer fits and the load fails.
• Twelve heads are not a power of two: forcing the closed form, or sorting or reversing the slopes, changes the vectors silently. small-en's eight heads cannot catch the first two, so verify on a twelve-head checkpoint.
• The slopes come from the code a checkpoint's own auto_map names; the canonical slopes on jinaai/jina-reranker-v1-tiny-en leave every score finite and wrong.
• The fused feed-forward gates its first half in the base and QK-norm shapes and its second half in the split-norm shape.
• F16 on the CPU returns NaN for base-de on every input and for base-zh on some, short inputs included, so the builder refuses F16 on the CPU for both; use F32 there, or F16 on an accelerator.
• A whitespace-only input is not empty on the byte-level tokenizers: six tokens on base-de and base-es, four on base-code.

References