Cross-Encoder Reranking

A reranker retrieves nothing. It is a scoring function that takes a query and one document and returns one number, and it has to be called once per document. Everything strange about reranking — the cost, the depth parameter, the fact that adding one sometimes makes your results worse — follows from that single sentence.

The thing in your vector index is a bi-encoder: a model that turns one piece of text into one vector, on its own. It saw your chunk at index time, months before anyone typed a question, and it had to compress that chunk into a single point that would have to serve every query anyone would ever ask. Retrieval is then a geometry problem — find the nearest points to the query's point — which is why it is fast.

A cross-encoder refuses that bargain. It takes the query and the document, joins them into one input, and runs them through the network together, so every query token can attend to every document token and the document's internal representation is allowed to depend on what you asked. The output is not an embedding. It is one relevance score for that one pair. There is nothing to store and nothing to index.

Below is a 25-chunk internal handbook with both models running on it. The left column is what the bi-encoder returns; the right column is what comes back after the joint scorer reorders the candidates it was handed. Start with the default query and move candidates sent to the reranker first.

Read the default state. The answer to "p99 latency spike after adding a read replica" is handbook#replica-notes, a page about read replicas that spends most of its length on storage billing, snapshot retention, access reviews and vault secrets, and one sentence on the thing you asked about. The bi-encoder puts it seventh, at 0.627, behind guide#latency-budget at 0.857 — a page that defines the latency alerting threshold and has nothing to say about replicas.

That is not the embedding failing. It is the embedding doing exactly what it was asked to do. Mean pooling gave the one useful sentence 21 of the chunk's 81 tokens, so it contributed 25.9% of the vector and the billing sentences contributed the rest. The chunk's single point in space is a compromise between seven topics, and it had to be struck before your query existed.

Now look at the attention panel. At β = 6, sentence 6 — "A new replica serves reads with a cold cache, so the p99 latency spike on replica reads lasts about an hour" — carries 94.8% of the weight, because its cosine against this query is 0.941 and the next best sentence in the chunk manages 0.381. The chunk's vector is recomputed around that sentence and the score goes from 0.627 to 0.936. It moves to rank 1. Nothing was retrieved; one of the ten candidates was re-described.

Then drag candidates sent to the reranker down to 5, and watch the answer vanish. Not drop — vanish. It was seventh.

Why the score cannot be precomputed

Set β to 0 and compare the two columns. They are identical, to every decimal the panel prints and eleven more it does not. The joint scorer with its attention switched off is the bi-encoder. Everything a cross-encoder can do that a bi-encoder cannot is contained in the freedom to weight the document differently depending on the query.

And that freedom is precisely what makes it uncacheable. The bi-encoder's output for a chunk is a function of the chunk alone, so you compute it once and store it. The cross-encoder's output is a function of the pair, so there are as many outputs as there are pairs. The sentence-transformers documentation puts the consequence bluntly: "A Cross-Encoder does not produce a sentence embedding. Also, we are not able to pass individual sentences to a Cross-Encoder." Their own worked example is clustering 10,000 sentences — about 50 million pairs, roughly 65 hours with a cross-encoder, against 5 seconds of bi-encoder embedding followed by ordinary vector arithmetic.

The original passage reranker, Nogueira and Cho's 2019 "Passage Re-ranking with BERT", is a two-page description of doing exactly this. They "feed the query as sentence A and the passage text as sentence B", take the [CLS] vector through a single linear layer to get the probability that the passage is relevant, and — the load-bearing sentence — "compute this probability for each passage independently". It took the top spot on the MS MARCO leaderboard, 35.8 MRR@10 on the evaluation set against the previous best of 28.1, which is the 27% relative improvement the paper is remembered for.

Two details from that paper are worth carrying into your own pipeline, because they are still true of every cross-encoder API you can buy. The query is truncated to 64 tokens. The query, passage and separators together are truncated to 512. If your chunks are pages, the reranker is scoring the first part of the page and guessing about the rest, which is one more reason that where you cut the document decides what the rest of the pipeline is allowed to do.

The arithmetic that forces it to be a second stage

Khattab and Zaharia measured it in the ColBERT paper. Reranking the official BM25 top-1000 for a single MS MARCO query with BERT-base took 10,700 ms and 97 teraFLOPs of arithmetic on one Tesla V100. BERT-large took 32,900 ms and 340 teraFLOPs. ColBERT, which keeps a vector per token and scores with a cheap interaction instead of a joint forward pass, took 61 ms and 7 gigaFLOPs — 175 times faster and about 14,000 times less arithmetic — for 34.9 MRR@10 against BERT-base's 34.7.

Divide 10,700 by 1,000 and you get the number this whole architecture is built around: 10.7 ms per query-document pair. That is the figure the panel's cost model uses, and it is why the readouts behave the way they do. Ten candidates is 107 ms. A hundred is 1.07 seconds. MS MARCO's collection is 8,841,823 passages, so scoring all of them for one query is about 26 hours. Drag a real corpus, for the cost projection and watch the number cross from "a latency budget" into "a batch job" somewhere around ten thousand chunks.

Be careful how you carry that number around. It is a 2020 measurement of BERT-base on a V100, and a 2026 reranker is a smaller distilled model on faster hardware with better batching, so the absolute figure is long out of date. The shape is not. Cost is linear in the number of candidates, it is paid on every query, and it cannot be amortised across queries the way an index can, because the thing being computed depends on the query. No hardware generation changes that; it is the definition of the model.

The middle of the panel's β slider is, loosely, where late interaction lives: enough query-dependence to fix the pooling problem, cheap enough to run wide. That is a real architectural family and it deserves its own page.

The candidate window is a ceiling, and it defaults to 10

Put the query back on the replica question, keep β at 6, and set the candidate slider to 5. The final list's top result is runbook#tail-latency at 0.869 and the answer is not in the list at any position. The scorer that would have ranked it first at 0.936 was never given it. Tick score every chunk in the corpus instead and it returns to rank 1 — the answer was always findable, it was always scorable, and the only thing standing between you and it was a number in a config file.

That number is smaller than most people think. Elasticsearch's text_similarity_reranker retriever takes a rank_window_size parameter described as "the number of top documents to consider in the re-ranking process", and it defaults to 10. If you added a reranker to an Elasticsearch query and did not set that parameter, your expensive cross-encoder is reordering ten documents.

Two consequences follow, and they point in opposite directions from what people expect.

Recall is fixed before the reranker starts. Whatever fraction of your queries have their answer inside the window is the best the whole pipeline can do, and no reranker improves it by a single query. ColBERT's Table 2 gives the numbers for BM25 on MS MARCO: recall of 59.2 at depth 50, 73.8 at 200, and 85.7 at 1000. Read those as ceilings. Rerank a window of 50 and four queries in ten are unanswerable no matter how good your reranker is. Even at a window of 1000 — ten seconds of V100 time per query — one query in seven has no correct document in the set being reordered. Nogueira and Cho say the same thing about their own setup in one clause: "some of the relevant passages might not be retrieved by BM25."

And your window is not even the true top-k. If the first stage is an approximate index, it has already dropped candidates before the window is applied, so the effective ceiling is lower than the exhaustive recall figure you looked up. That is the recall of the approximate index, and it composes multiplicatively with everything here. The only stage that can genuinely raise recall is widening the retrieval, which is what running a second retriever and fusing the lists does: fusion changes the candidate set, reranking changes its order. Only one of them can help a query whose answer is at rank 340.

The two ways a reranker makes things worse

One: it promotes the page that restates your question. Switch the query to "connection pool exhausted during a rolling deploy". The answer, handbook#ops-misc, is at bi-encoder rank 4. At β = 6 the reranker moves it up — to rank 2, at 0.974, behind handbook#faq-index at 0.998.

Look at what handbook#faq-index is. It is a table of contents. One of its lines reads "Connection pool exhausted during a rolling deploy: see the ops handbook." That line is a near-perfect match for the query because it is the query, and once the scorer is allowed to concentrate, it concentrates there and scores the chunk on that line alone. The bi-encoder never had this problem: mean pooling diluted that one line across seven other lines about memory limits, index bloat and billing, and put the index page at rank 8.

This is the mechanism behind every report of the form "my reranker keeps promoting the FAQ / the glossary / the changelog / the docs index". Resemblance to the query and answering the query are different properties, and a scorer with sharper attention gets better at measuring the first one. Drag β down: at 3 the answer is first at 0.9728 against 0.9540, at 2 it is first at 0.9677 against 0.8513, and at 4 it loses to 0.9861. The useful band is narrow and nothing in the API tells you where it is.

A production cross-encoder handles this case better than the panel's one-layer cartoon does, because it was trained on query-passage relevance judgements and has therefore seen many examples of a title that restates a question and does not answer it. Better, though, is not solved: an unanswered Stack Overflow question titled "Azure AI Search semantic ranker degrades vector search quality" is exactly this failure met in production, and the reason nobody answered it is that the answer requires measuring the stage separately rather than reading a config page.

Two: it changes nothing and you pay for it anyway. Switch to "does raising max_connections fix pool exhaustion". The bi-encoder already returns guide#max-connections first at 0.998, because that chunk is short, on-topic in every sentence, and has nothing to dilute. Move β across its entire range. The answer stays at rank 1 and the top of the list never changes. You have added 107 milliseconds to the critical path, before generation has started, to confirm a decision that was already correct.

Both failures share a shape: a reranker is a precision instrument, and most RAG pipelines that feel broken are broken on recall. Before you add one, measure how often the answer is already in the window and how often the stage-1 top-1 is already right. Those two numbers tell you the maximum and the minimum benefit available, and they take an afternoon to produce.

There is one more control worth trying here, because it is genuinely counterintuitive. Keep the pool-and-deploy query at β = 6 and pull the candidate window from 10 down to 7. The answer jumps to rank 1. Narrowing the window did not improve the reranker; it removed the distractor, which sat at bi-encoder rank 8. Tuning rerank depth against a small evaluation set will find effects like this and present them as improvements. They do not generalise, because they are facts about which distractor happened to sit just inside the boundary.

Checking it on a real system

Every diagnosis in this lesson is the same measurement: instrument the two stages separately and record the rank of a known-good document in each.

Measure the ceiling before you measure the reranker. Take fifty queries with the correct chunk labelled, run only stage 1, and report the fraction of queries whose answer appears within your candidate window. That is recall at k where k is your rerank depth, and it is the reranker's maximum achievable top-1 rate. If it is 0.62, then 38% of your queries are lost before the reranker is called and the reranker is not the thing to fix. Report it at several depths — 10, 50, 200 — and the shape of that curve tells you what widening the window is worth in your corpus, which is a completely different number from what it is worth in MS MARCO.

Elasticsearch. Check whether rank_window_size appears in your text_similarity_reranker block at all; if it does not, it is 10. Set it well above size — the examples in Elastic's own documentation use 100 — and note that raising it multiplies your inference bill and your latency linearly, so this is a budget decision with a quality side effect rather than the reverse. The retriever also accepts min_score, and the documentation's warning about it is the important part: "score calculations vary depending on the model used." Which brings us to the next thing.

Do not threshold or alert on cross-encoder scores. A cross-encoder emits a per-pair logit or probability from a model trained on a particular relevance distribution. It is not calibrated across queries, it is not comparable across models, and it changes meaning when you change the model version. A cutoff of 0.5 that drops 3% of results today can drop 40% after a model upgrade with no code change and no error. If you need a cutoff, derive it per deployment from a labelled set and re-derive it whenever the model changes.

Log both ranks, per query. For every request, record the stage-1 rank and the final rank of whatever the user ended up clicking or the answer ended up citing. Three buckets fall out and each has a different fix: the answer was outside the window (widen retrieval, or fuse a second retriever — the reranker is irrelevant to this bucket); the answer was inside the window and the reranker demoted it (your distractors resemble the query more than your answers do, which is usually a chunking or a boilerplate problem); the reranker left the order alone (you are paying for nothing on these queries and could route them past it).

Budget the depth twice. The candidate window costs you inference time before generation starts, and the documents you keep afterwards cost you again in the KV cache during generation. Those are the same slider from two directions, which is the actual argument for the funnel shape: retrieve wide, because recall is set there and cannot be recovered; rerank a window you have measured; pass five good chunks to the model rather than twenty mediocre ones, because prefill is charged per token and attention quality falls off with irrelevant context.