Hybrid Retrieval and Reciprocal Rank Fusion

Reciprocal rank fusion does not average your BM25 score and your cosine score. It throws both of them away and keeps only the positions. That is the entire idea, it is the reason hybrid search works at all, and it is also the reason hybrid search sometimes returns a worse answer than either retriever did alone.

The two retrievers fail in opposite directions, which is why anyone bothers combining them. A dense retriever maps text into a learned space where "money back" lands near "refund" — and where ERR_CONN_2043 lands near ERR_AUTH_1180, because a subword tokenizer shreds both into fragments the encoder never saw in training and they end up pointing in almost the same direction. BM25 has no notion of meaning at all, but it knows that ERR_CONN_2043 occurs in exactly one document out of fourteen, which is the single most useful fact available about that query.

Below is a fourteen-chunk support corpus with both retrievers running on it. BM25 is Lucene's, formula for formula. The dense side is real vector arithmetic over a synthetic term space. Start with the query ERR_CONN_2043 retry and read the two lists before you touch anything.

Read the dense column for ERR_CONN_2043 retry. Rank 1 is faq#retry — "Should I retry a failed request?" — which is a question, not an answer, and contains no error code. Rank 2 is runbook#auth, the runbook for ERR_AUTH_1180: a different error, a different subsystem, a different fix. The chunk that literally contains ERR_CONN_2043 is third. The embedding is not broken; it simply has no dimension in which those four error codes differ. Their pairwise cosines in this space are 0.96 to 0.98 — an encoder that has never seen a token cannot distinguish it from another token it has never seen.

Now read the BM25 column. One term, err_conn_2043, appears in 1 document of 14, so its IDF is ln(1 + 13.5/1.5) = 2.303 against 1.455 for retry, which appears in three. That single term contributes 0.957 of the chunk's 1.561 total and puts the right answer first, decisively, using no semantics whatsoever.

And the fused list, at the default k = 60, puts faq#retry first — score 0.03252 against the correct chunk's 0.03227. Hybrid retrieval, with both retrievers behaving exactly as designed, is worse here than the lexical retriever was on its own. The rest of this lesson is about why, and about what the checkbox at the bottom right of the panel does to it.

Why not just add the scores

Switch how the two lists are combined to weighted sum of scores and look at what happens across the four queries. On this query it fixes the problem: 2.082 for the right chunk against 1.716, a clean win. Now switch the query to the paraphrase, how do I get my money back. The fused top hit scores 8.227 and it is faq#refund — "How do I get my money back? See the refunds page." — a link, not an answer. The chunk that actually states the refund policy scores 0.938.

Those numbers are not comparable and the sum does not care. BM25 gave faq#refund 7.255 because a short document matched six query terms; the dense retriever gave it 0.972 because cosine cannot exceed 1. Adding them produces 8.227, of which 88% is BM25's opinion. The weighting you chose is not the weighting you got. BM25's scale depends on the length of the query, the length of the document, the size of the corpus and the document frequency of every term in it — change any of those and the effective weight changes with no config edit. Min-max normalising each list before summing only moves the problem: the maximum of a list is whatever the top hit happened to score, so on a query where nothing matches well you normalise noise up to 1.0.

Cormack, Clarke and Büttcher's 2009 paper proposed the alternative in one line:

RRFscore(d) = ∑_{r ∈ R} 1 / (k + r(d)), with k = 60

Their stated advantage is the one people skip when they describe RRF as a weighted average: it "combines ranks without regard to the arbitrary scores returned by particular ranking methods." Qdrant's implementation says the same thing in a code comment — "the input scores are irrelevant, only the order matters." A retriever that returns 7.255 and one that returns 0.972 are put on the same footing because neither number is used.

That immunity is bought with information loss, and the loss is exactly what hurt us on the first query. In the BM25 list, rank 1 scored 1.561 and rank 2 scored 1.031 — a 51% gap that says the top hit is not merely first but far first. RRF sees 1 and 2. Meanwhile faq#retry is rank 2 in BM25 and rank 1 in dense; the right chunk is rank 1 in BM25 and rank 3 in dense. Compare the two sums term by term and the outcome is settled before you pick a k:

faq#retry  = 1/(k+1) + 1/(k+2)
runbook#conn = 1/(k+1) + 1/(k+3)

The first terms are identical and the second differs only in the denominator, so faq#retry wins for every value of k. Drag the rrf rank constant from 0 to 100 and watch the fused order refuse to change. This is worth knowing before you spend an afternoon tuning k: when one document dominates another on ranks, no rank constant will reverse it, because k is a monotone reparameterisation and not a re-ranking.

What k actually controls

k sets how quickly the value of a position decays. At k = 0 the top hit is worth 1.0 and the tenth is worth 0.1 — a factor of ten. At k = 60 the top hit is worth 0.0164 and the tenth 0.0143 — a factor of 1.15. The paper is candid about how it landed on the number: k = 60 "was fixed during a pilot investigation and not altered during subsequent validation", and their own table shows MAP moving from 0.2072 at k = 0 to a plateau of about 0.2145 across k = 50 to 100, "near-optimal, but... not critical."

So k is a smoothing constant, not a relevance knob. What it decides is how much weighting can do. Leave the query on the error code and drag weight on the lexical list to 2.0:

• At k = 60 the right chunk takes first place at 0.04866 against 0.04865 — correct by eight parts in a million. A single document entering or leaving either list would undo it.
• At k = 2 the same weight gives 0.8667 against 0.8333 — a 4% margin, visible, stable, and meaning the same thing.

The margin readout makes this visible without arithmetic. Both settings give the same ordering; only one of them gives an ordering you could deploy. This is why the two production systems you are most likely to use disagree by a factor of thirty on the default. Elasticsearch's RRF retriever ships rank_constant: 60, following the paper. Qdrant's query API ships DEFAULT_RRF_K = 2, with the comment "mitigates the impact of high rankings by outlier systems" — and it counts positions from zero with a shifted formula, so its k = 2 is the paper's k = 1. Neither is wrong. But if you tune weights against one and migrate to the other, every weight you picked now does something different, and nothing in either API will tell you.

The boundary. Set k to 0. Now only the top of each list has any real value, so a retriever that is confidently wrong about its first result — as the dense side is here — has maximum influence. Set k to 100 and every candidate in both lists is worth nearly the same, which turns fusion into an approximate popularity vote over the union of the two candidate sets. Both ends are degenerate. The plateau between them is wide, which is the honest version of "RRF requires no tuning": it does not need tuning because tuning it does not do very much.

The candidate window is a trapdoor

Set the query to the paraphrase and pull candidates taken from each list down to 1. The answer disappears from the fused list entirely — not demoted, absent. It is unranked in BM25 and rank 2 in the dense list, so depth 2 is the shallowest setting that includes it at all; one notch further down and nothing in the pipeline can know it ever existed.

This is a real default in a real system rather than a hypothetical. Elasticsearch's rank_window_size "defaults to the size parameter", so a search asking for 10 results reads only 10 from each child retriever before fusing. The documentation is explicit that rank_window_size is "all the results that we'll get to see from the individual query components". A document sitting at rank 12 in both lists — which is a completely ordinary place for the right answer to sit when the query is hard — is not ranked low by fusion. It is not an input to fusion. Setting it well above your size is close to free, and leaving it at the default is the most common silent recall bug in a hybrid stack.

The same slider caps the re-scorer, and there it costs more. Turn on re-score the candidates jointly, keep the paraphrase query, and step the depth up from 2: at 2 the answer is still missing, at 3 it appears at rank 2, and it stays there. A reranker cannot recover a document the retriever never returned. It is a precision instrument bolted onto a recall problem, which is why "we added a reranker and it did not help" is usually a report about the retrieval stage.

What the joint scorer sees that neither retriever can

Go back to ERR_CONN_2043 retry and tick the re-score box. The right chunk moves from rank 2 to rank 1, and the margin goes from 0.8% to about 2.5%. Nothing was retrieved that was not already there; the same candidates were put in a different order by a scorer that compares the query and the chunk term by term instead of comparing two averages.

That is the whole architectural difference, and it is worth stating precisely because "reranker" gets used for three different things. A bi-encoder — the thing that filled your vector index — must commit each chunk to a single vector before any query exists. That is what makes it fast: the vectors are computed once, offline. It is also what makes it lossy, and the mean-pooled vector is where the information goes. A cross-encoder does the opposite. It concatenates the query and the chunk and runs them through the network together, so every query token can attend to every chunk token and the output is a single relevance score for that specific pair. Nothing can be precomputed, so the cost is one forward pass per candidate — which is exactly why it runs on 50 documents and not on 5 million. Nogueira and Cho's 2019 "Passage Re-ranking with BERT" was a straightforward implementation of that idea and it improved MS MARCO MRR@10 by 27% relative over the previous state of the art.

The re-scorer in the panel is the middle option, ColBERT's late interaction: it keeps a vector per term, and scores a pair as the sum over query terms of the best-matching chunk term. Khattab and Zaharia's argument for it is architectural — delay the interaction, but keep it fine-grained, so document representations can still be precomputed offline. In the panel it is enough to fix the error-code query, because the query's identifier term is compared against the chunk's identifier term directly, instead of both being averaged into a mean that no longer distinguishes them.

Where it fails. Set the query back to the paraphrase, with re-scoring on and depth at 10. The answer is rank 2, at 0.996, behind faq#refund at 1.000. Every configuration in this panel leaves it at rank 2. The FAQ entry restates the user's question in the user's own words, so it is the closest match by lexical overlap, by pooled cosine and by term-level alignment simultaneously — all three measure resemblance, and it resembles the query more than the answer does. Only a scorer trained on query-passage relevance judgements, which has learned what an answer looks like as distinct from what a restatement looks like, demotes it. That training signal is what you are buying from a cross-encoder, and it is not something late interaction reproduces.

Do not read the last three sections as "dense retrieval is bad". Karpukhin et al. established the opposite in 2020: a dual-encoder trained on question-passage pairs beat a strong Lucene BM25 baseline by 9 to 19 points absolute on top-20 retrieval accuracy. The complication arrived with generalisation. BEIR evaluated ten retrieval systems across eighteen datasets and concluded that "BM25 is a robust baseline" while dense models "often underperform" out of domain; Sciavolino et al. built a set of simple entity-centric questions and found dense retrievers "drastically underperform sparse methods", generalising "only to common entities unless the question pattern is explicitly observed during training". Your product names, error codes, SKUs and internal acronyms are rare entities by construction. That is not a defect you can fine-tune away cheaply, and it is the reason the lexical list stays in the pipeline.

Checking it on a real system

Every diagnosis here is the same shape: get the two lists separately, before fusion, and compare positions rather than scores.

Elasticsearch. Run each child retriever as its own query and record the rank of the known-good document in both. Then check rank_window_size — if it is absent from your request body it is equal to size, and that is your real recall ceiling. Use _explain on the standard retriever to see the BM25 breakdown; the explanation names the parts directly, including "k1, term saturation parameter" and "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))". If a term you expected to dominate has a low IDF, your analyzer split it — check whether ERR_CONN_2043 survives your tokenizer as one term or arrives as err, conn, 2043, in which case its rarity is gone and BM25 cannot help you either.

Qdrant. A prefetch with fusion: rrf returns fused scores; run the prefetches as top-level queries to see the inputs. Note the default k is 2 and the weights are per-prefetch, and that fused scores are not comparable to either input's scores or to the fused scores of a differently-configured collection. Do not alert on them.

Postgres with pgvector. There is no fusion primitive; you write the RRF join yourself, usually two CTEs with row_number() and a sum of 1.0 / (60 + rn). Two things bite here. The lexical side is exact and the vector side is not — hnsw.ef_search defaults to 40, so an HNSW scan returns at most 40 candidates before your LIMIT applies, and ivfflat.probes defaults to 1, which reads a single list out of however many you built. Both silently return fewer or worse rows rather than erroring. And filtering is applied after the index scan, so, in pgvector's own words, "if a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average". A tenant filter on a hybrid query can quietly reduce your dense candidate list to nothing. That whole failure mode is the recall of the approximate index, and it sits underneath everything in this lesson.

The measurement that decides it. Build a set of fifty real queries with the correct chunk labelled — deliberately including the ones with error codes, part numbers and internal names, because those are the queries that separate the two retrievers — and report recall@k four ways: dense alone, lexical alone, fused, and fused-then-reranked. You are looking for the queries where fused is worse than the better of its two inputs. Those exist in every corpus, they are the ones this lesson is about, and averaging them into a single nDCG hides them completely. If a large fraction of your queries are exact-identifier lookups, routing those to the lexical retriever and skipping fusion for them is a legitimate answer.

Two costs to keep in view. A cross-encoder is a full forward pass per candidate, so reranking 100 chunks is 100 sequence-length-limited inferences on the critical path before generation starts; the depth slider is a latency budget as much as a quality knob. And every chunk that survives to the prompt is paid for again in the KV cache during generation, which is the real argument for reranking down to five good chunks rather than passing twenty mediocre ones. Neither of those is a reason to skip the stage — they are the reason the stage is a funnel: retrieve wide, fuse, rerank narrow. Whether a chunk deserves to be in that funnel at all is decided earlier still, by where you cut the document.