Cosine Similarity Is Not Relevance

Your retriever does not rank chunks by how likely they are to answer the question. It ranks them by the angle between two averaged vectors, and the chunk with the smallest angle is very often the one that restates your question instead of answering it.

Cosine similarity is a well-defined quantity and it is computed correctly. That is the problem. It answers how alike are these two texts, in the geometry this encoder was trained to produce — and most encoders were trained on sentence-similarity and paraphrase data, where two texts are "similar" when one could be substituted for the other. A question and its answer are not substitutable. They are, in the encoder's own terms, dissimilar in exactly the way it was taught to notice.

Underneath that there is a second, cruder mechanism that does most of the damage in practice. Almost every embedding model turns a passage of many tokens into one vector by averaging. Sentence-BERT's original published pooling is mean pooling, and sentence-transformers/all-MiniLM-L6-v2 — still the default in half the tutorials — ships pooling_mode_mean_tokens: true in its 1_Pooling/config.json. An average has no way to say "one part of me is exactly what you asked for". Attach nine sentences of legal boilerplate to the one sentence that holds the answer and the answer's contribution to the mean drops to a tenth.

The panel below is a retriever, built out of real vector arithmetic. Eight chunks, an eleven-dimensional space, mean pooling, L2 normalisation, cosine. Start with boilerplate appended to the answer's chunk at zero and drag it to one.

At zero boilerplate the retriever is perfect: returns-policy #window comes back first at 0.904, comfortably ahead of the two decoys. Add one sentence — "This policy is governed by the laws of the State of Delaware." — and it falls to rank 3 at 0.822. Nothing about the answer changed. Nothing about the query changed. Nothing about the other seven chunks changed. One sentence of unrelated text, appended to the paragraph that holds the answer, moved the answer's vector far enough that two chunks which do not contain the answer now sit above it.

Push on to nine and the top of the list is worth reading carefully. First is returns-policy #processing at 0.870: refunds are processed within five business days. It is about refunds, it contains a number of days, and it answers a different question. Second is faq-index at 0.865, which is the sentence "How do I return an item? How long do I have?" — your own question, reflected back, containing no answer whatsoever. The chunk that actually says "30 days" is now sixth, at 0.613, and with top_k = 5 the model never sees it. The model will then answer confidently from the five chunks it did get, which is how a retrieval bug arrives at the user as a hallucination.

The averaging is the mechanism

Write it out. A chunk of n sentences with vectors s₁ … s_n is embedded as their mean, then normalised:

e(chunk) = (s₁ + s₂ + … + s_n) / n, then divided by its own length

The sentence that answers the question is one term in that sum. Its weight in the result is 1/n and nothing else — not its salience, not its position, not whether it is the only sentence in the document a human would call relevant. At n = 1 the chunk vector is the answer's vector. At n = 10 the answer contributes 10% of the direction and the other 90% points wherever the surrounding text points, which in a policy document is straight at boilerplate.

The log reports this as an angle. Set the slider to 9 and read the dilution line: the chunk's vector has rotated 48.7° away from the vector of the sentence inside it that answers the question. Cosine at 48.7° is 0.66, so a third of the available similarity is gone before the query is even considered. You have not lost the answer — it is still in the chunk, the text is still there, and if you print the chunk you will see the words "30 days" — but the object the index compares against the query no longer points at it.

This is why "just use bigger chunks so the context is complete" and "just use smaller chunks so retrieval is precise" are both correct and in direct opposition, and why fixed-size chunking is a trade rather than a setting. Larger chunks raise the chance that a chunk contains the answer and lower the chance that it ranks. There is no chunk size at which both are maximised, which is the entire reason the small-to-big and parent-document retrievers exist: index the small vector, return the large text.

Note what the slider does not do. Watch the "what comes back first" readout while you drag it from 1 to 14: it stays on returns-policy #processing at 0.870 the whole way. The score of the wrong answer is a constant. Only the right answer's score moves. If you were monitoring top-1 similarity as a retrieval health metric, this entire failure is invisible to it.

Everything scores 0.8, and that is not a bug

Put the boilerplate slider back to 9 — the last section walked it to 14 — then set shared component to 0 and then to 2.0, and watch the second panel. At 0 the eight scores spread across 0.00 to 0.71. At 2.0 they are crushed into 0.83 to 0.95 — every chunk in the corpus, including the product page about widget colours, is now more than 80% similar to a question about returns.

Real embedding spaces behave like the right-hand end of that slider. Learned embeddings are anisotropic: they occupy a narrow cone rather than filling the sphere, so a large component is common to every vector in the space and every pair of unrelated texts still scores well above zero. The cone is a property of the model, not of your data, and it is the reason the question "what similarity threshold should I use?" has no transferable answer. 0.8 is a strict cut-off for one model and permissive for the next.

Now look at the ranks while you move that slider. At 9 sentences of boilerplate they do not change: the answer sits at rank 6 and returns-policy #processing sits at rank 1 at every one of the twenty-one positions. The shared component moved every score by half a point and moved no ordering at all. This is worth internalising, because it splits the diagnosis in two: a compressed score band is a calibration problem, and the answer being at rank 6 is a ranking problem, and they have nothing to do with each other.

That invariance is a near-miss rather than an identity, and the panel will show you the miss. Adding the same vector to every embedding shifts each one by the same amount, but cosine divides by each vector's own length afterwards, and that division is not the same for a long chunk vector as for a short one. So the ordering can move. Put the boilerplate at 14 and drag the shared component up again: the answer swaps rank 6 for rank 7 as you pass 1.20, and stays there. Eight of the fifteen boilerplate settings have a swap like that somewhere on the slider. The size of the effect is the point: over that same range the answer's score moves from 0.142 to 0.853, and its rank moves by at most one place at any setting. So "anisotropy is a calibration problem, not a ranking problem" is a good working rule and not a theorem. Put the boilerplate back to 9 before going on.

Put the shared component back to 1.00 — the paragraph above left it at the top — and tick subtract the corpus mean first to see the split cleanly. Centring removes the shared direction: the band opens from 0.30 wide to 0.98 wide, and the scores become genuinely informative — negative for the chunks that are about something else. And the answer moves from rank 6 to rank 7. The calibration is fixed and the retrieval is marginally worse. A better thermometer does not lower the fever.

Untick the centring again and switch score function to raw dot product. The ranking changes again: faq-index takes first place, the answer moves to rank 5. Nothing about the texts changed; the ordering is a property of the choice between normalising and not. Steck, Ekanadham and Kallus made the sharp version of this argument in 2024 — for regularised linear models they derive closed forms showing cosine similarity "can yield arbitrary and therefore meaningless similarities", and that for some models the value is not even unique but is implicitly set by the regularisation used during training. In deep models the regularisation is a mixture of effects nobody enumerated, so the cosine you read out has no principled scale. It is an ordering device, and a fragile one.

Asymmetry: the query is not shaped like the answer

Put score function back to cosine — the section above left it on the raw dot product — and change what the user sends to the hypothetical answer. With the boilerplate at 9 the answer climbs from rank 6 to rank 4; drop the boilerplate to 0 and it goes to rank 1 at 0.999. The corpus is identical. The only difference is that the query is now a declarative sentence shaped like the thing it is looking for.

That is the whole idea behind writing a hypothetical answer and searching with it, and it works for a mechanical reason: the encoder places texts by surface form as well as topic, and a question sits near other questions. The same effect is what put faq-index — pure interrogative, no content — at rank 2.

The sentence-transformers documentation has drawn this distinction for years and it is routinely skipped. It calls the two cases symmetric semantic search — query and corpus entries are the same length and kind, so you could swap them — and asymmetric semantic search, "a short query… and you want to find a longer paragraph answering the query", where "flipping the query and the entries in your corpus usually does not make sense." Different models, different training data. A model trained on Quora duplicate-question pairs is a paraphrase detector; pointing it at a question-to-passage task is a category error that produces plausible nonsense rather than an exception.

Some models make the asymmetry explicit and are used wrongly anyway. intfloat/e5-base-v2 requires the literal strings "query: " and "passage: " to be prepended, and its model card is unambiguous about which goes where: "query: " and "passage: " for asymmetric tasks like open-QA passage retrieval, "query: " on both sides for symmetric ones. Omit the prefixes, or use the same prefix for both sides of a retrieval task, and the model still returns vectors, still returns numbers between 0 and 1, and quietly ranks worse. sentence-transformers has exposed encode_query and encode_document as separate calls since v5.0.0, for the same reason, and its own semantic-search guide recommends them for asymmetric search; if your indexing job and your query path both call plain encode, check whether your model wanted otherwise.

Where this fix stops. Set the hypothetical-answer query and push the boilerplate back to 12. The answer is at rank 5 again. Rewriting the query changes which direction you search in; it does nothing about a chunk vector that has been averaged away from its own contents. Query-side fixes cannot repair index-side dilution, and most RAG tuning effort goes into the query side because that is the side you can change without a re-index.

Checking it on a real system

The diagnosis is cheap and almost nobody runs it. Take twenty real questions, label the chunk that actually contains each answer, and record the rank of that chunk — not whether the final answer looked right. Three numbers fall out, and they point at three different bugs:

• The answer's chunk is not in the index at all. An ingestion or parsing bug. Nothing downstream can help.
• It is in the index and its rank is worse than your top_k. This lesson. Retrieval recall is your ceiling — recall@k is the number to track, and no prompt engineering raises it.
• It is inside top_k and the answer is still wrong. A generation or ordering problem, not a retrieval one. Worth knowing that position inside the context matters too: Liu et al. found model accuracy "significantly degrades when models must access relevant information in the middle of long contexts", so the chunk at rank 4 of 5 is not being read as attentively as the chunk at rank 1.

Then check the two things about your encoder that are stated in its own files and almost never read.

Its real input limit. sentence-transformers/all-MiniLM-L6-v2 has max_seq_length: 256 in sentence_bert_config.json, while the BERT backbone underneath it reports max_position_embeddings: 512. If you sized your chunks at 512 tokens because that is what the architecture supports, every chunk is being silently truncated at 256 word pieces and the second half of each one has never been indexed. There is no warning and no error; truncation is the library's normal behaviour. Print model.max_seq_length and len(model.tokenizer(chunk)["input_ids"]) for your longest chunk before you trust anything else.

Its pooling. 1_Pooling/config.json says which one: all-MiniLM-L6-v2 uses mean pooling (pooling_mode_mean_tokens: true) and BAAI/bge-base-en-v1.5 uses the CLS token (pooling_mode_cls_token: true). They dilute differently, and neither escapes the trade — CLS is one position's output, not an average, but it is still a single vector that has to stand in for the whole passage before any query is known.

Finally, sanity-check the geometry itself with three lines: embed 200 random unrelated chunks from your own corpus, take all pairwise cosines, and look at the distribution. If the 5th percentile of unrelated pairs is 0.75, your live "0.82 similarity" hit is at the 30th percentile of noise, and any threshold you have deployed against it is superstition. That histogram, not a number from a blog post, is where a threshold comes from — if you use one at all.

None of the above is a fix. The fixes are structural, and they are the next two things to read: retrieve with a lexical signal alongside the vector one so an exact token can win, then re-score the candidates with a model that reads the query and the chunk together instead of comparing two averages — that is hybrid retrieval and reranking, and it is the direct answer to this lesson. Separately, be aware that your vector index is not even returning the true top-k: HNSW and IVF are approximate, and the recall of the index itself is a parameter you have probably left at its default. And whatever you retrieve has to be paid for in context, where every token is a real cost in the KV cache — raising top_k to cover a ranking problem is the most expensive way to not fix it.