Query Rewriting and HyDE

HyDE does not improve your query. It throws your query away. An instruction-following model is asked to "write a paragraph that answers the question", the paragraph it invents is embedded, and the vector search runs on that instead. Your question never touches the index. Understanding why that helps — and exactly when it stops helping — is a fact about the shape of text, not about the quality of the question.

Start with the problem it solves. A question and its answer are written differently. "How do I stop getting 429 errors?" is interrogative, short, and contains the user's confusion. The reference page that answers it is declarative, states a rule, and never uses the word "how". An embedding model trained without relevance labels — trained, that is, on whether two pieces of text are alike — puts your question next to other questions. That is why vector search so often returns the FAQ entry that restates what you asked, sitting above the page that answers it.

HyDE — hypothetical document embeddings — closes that gap by making the search text document-shaped. Its authors say so plainly: it "circumvents the aforementioned learning problem by performing search in document-only embedding space that captures document-document similarity." The generated paragraph does not need to be true. It needs to be the right shape.

Below is an 18-chunk API documentation corpus with reference pages, FAQ entries, forum threads and blog posts in it, each carrying a measured position between "written as a question" and "written as a statement". Turn HyDE on and off first, then go looking for the settings that make it lose.

Read the default. The query is "how do I stop getting 429 errors" and the answer is ref#retry-budget, which says to back off exponentially and gives the retry budget. The plain query returns faq#stop-429 first at 0.993 — an FAQ heading whose entire text is "How do I stop getting 429 errors?" — then a forum thread at 0.965, then another FAQ at 0.959. The answer is fourth, at 0.646.

Nothing is broken. Those three chunks are genuinely the most similar things in the corpus to what you typed, because what you typed is a question and they are questions. The retriever is doing similarity, and you wanted relevance.

Now tick HyDE. ref#retry-budget goes to rank 1 at 0.909 and the FAQ that restates your question falls to third. The search vector moved from the question corner of the space to the statement corner, and the ranking followed. No retriever was retrained, no chunk was re-indexed, and nothing in the corpus changed.

The gap it closes is a property of your encoder

Work the geometry, because it tells you in one step whether HyDE is worth trying on your system. Every text here is a topic direction of length √(1−g²) plus a style direction of length g. Take the cosine between a query and a document:

cos(query, doc) = (1 − g²)·cos(topics) + g²·cos(ask, style of doc)
cos(hypothetical, doc) = (1 − g²)·cos(topics) + g²·cos(tell, style of doc)

The topical term is the same in both lines. The difference is entirely the second term, and it is not a constant: it depends on the shape of each document. A question-shaped query gets a bonus of up to g² on question-shaped chunks and nothing on reference pages. A statement-shaped hypothetical gets the mirror image. That is the whole mechanism, and it explains a complaint you have probably made: your vector search keeps returning the FAQ.

Now drag how far this encoder separates questions from statements down to 0.00 with HyDE still on. The answer sits at rank 4 with HyDE and at rank 4 without it — the two columns become identical, because the second term is zero in both lines and only the topical term remains. HyDE has become an extra language model call that changes nothing. Bring it back up: at 0.35 the answer moves to rank 3, and at 0.50 it reaches rank 1 and stays there.

That slider is not a hyperparameter. It is a fact about the model in your index, and it has a name: an encoder trained on question-passage pairs, like Karpukhin et al.'s Dense Passage Retrieval, learns the two encoders jointly so that a question lands on its answer. It has been trained to have a small gap. An unsupervised contrastive encoder has not.

HyDE's own authors say the quiet part. Section 5.2 of the paper opens: "HyDE with fine-tuned encoder is not the intended usage: HyDE is more powerful and irreplaceable when few relevance labels are present." They ran it anyway, and found that "less powerful instruction LMs can negatively impact the overall performance of the fine-tuned retriever" — FLAN-T5-11b takes a fine-tuned Contriever from 62.1 to 60.2 nDCG@10 on TREC DL19. So the decision rule is short. If your retriever was fine-tuned on your query distribution, expect HyDE to cost you a language model call and buy nothing. If it is an off-the-shelf embedding model and your users type questions, expect a large win.

The hallucination is load-bearing

Turn how wrong each generation is up and watch what happens. This is not a defect the technique tolerates; it is the thing the technique is built around. The paper is direct: the generated document "is not real, can and is likely to be ungrounded factually. We only require it to capture relevance pattern." The encoder is expected to do the cleanup — "we expect the encoder's dense bottleneck to serve a lossy compressor, where the extra (hallucinated) details are filtered out from the embedding."

That is a real mechanism and it has a real limit, and the panel lets you find it. Set the query to the well-formed question, untick average the original query in too, push how wrong each generation is to 0.80, and step N through 1, 2, 4 and 8. The rank of the answer goes 1, then 6, then 4, then 2.

That sequence is the whole argument for sampling more than one document. At N = 1 the answer is first, and it is first by luck: that single generation drifted 0.314 and happened to land near the right neighbourhood. At N = 2 the two samples average 0.536 of drift and the top result becomes ref#billing, a page about invoicing. By N = 8 the mean drift of the individual generations is 0.557 — higher than at any smaller N — and yet the drift of their average is 0.429 and the answer is back at rank 2. The samples are not getting better. Their mean is, because they are wrong in different directions and the errors partly cancel.

The corollary is the uncomfortable one. With a single generation you cannot tell a good sample from a bad one, because the mean drift of one sample is that sample: the two drift readouts show the same number and there is nothing to compare it against. Equation 8 of the paper is what averaging looks like when you write it down:

v̂_q = ¹⁄_N+1 [ ∑_k=1..N f(d̂_k) + f(q) ]

Note what else is in that sum: f(q), the embedding of the original question. The paper "also considers the query as a possible hypothesis". It is a safety line — if every generation goes somewhere strange, the query drags the search vector part of the way back.

And notice the coefficient. The query gets weight 1/(N+1). At N = 8 that is 11% and the anchor is gentle. At N = 1 it is 50%, which means half of your search vector is the question you were trying not to search with. Tick average the original query in too, set N to 1, and set wrongness to 0: the answer is at rank 4, exactly where the plain query left it. HyDE with one generation and the query averaged in is HyDE half switched off. Either raise N or drop the query from the average; do not quietly do both halves of the wrong thing.

Where averaging cannot save you

Switch the query to "what is the default request timeout". The corpus has two chunks that could match: ref#request-timeout, which says 5 seconds, and ref#connect-timeout, which says 30. The model in this simulation has a firm belief that the answer is about connection timeouts — the same kind of belief a real model has about your product's defaults, your flag names and your version numbers, learned from a hundred other products' documentation and applied confidently to yours.

At the default wrongness of 0.25, HyDE still works: the answer goes from rank 2 to rank 1. Now set wrongness to 0.8. The top result is ref#connect-timeout at 0.914 and the answer has fallen to rank 4 at 0.603. HyDE retrieved the document that matches what the model believes instead of the document that answers the question, and it did so confidently, with a higher score than it ever gave the right answer.

Now the part that matters. Untick the query from the average and step N through 1, 2, 4 and 8. The rank does not move. Not by one position, at any wrongness setting. When every generation is wrong in the same direction, the mean of N of them is that same wrong vector, and the averaging that rescued the random case is arithmetically a no-op. The drift readout makes it explicit: drift per generation and drift after averaging are the same number.

Worse, with the query left in the average, raising N actively hurts here, because the only thing correcting the error is the query's 1/(N+1) share and you are shrinking it. Generating more hypothetical documents is a variance reduction technique. It does nothing about bias, and a language model's confident belief about your system is bias.

The practical form of this: HyDE is at its most dangerous on exactly the queries where you most need the corpus rather than the model — internal product names, version-specific behaviour, numeric defaults, error codes, and anything the model has seen a plausible-but-different version of elsewhere. A generated paragraph also paraphrases, which means a rare identifier in the original query can simply not survive into the text you search with. If your corpus depends on exact identifiers, keep a lexical channel that never sees the rewrite at all — run BM25 on the original query and fuse — and treat HyDE as the dense half only.

Rewriting is a different technique with a different failure

Switch to "does it work with the new one?" and leave both toggles off. This is a second conversational turn; on its own it means nothing. The retriever returns faq#new-version at 0.999 — an FAQ entry reading "Does it work with the new version?" — which is a perfect similarity match and a perfectly useless result. The answer, ref#retry-v3, is fourth at 0.452.

Now try the three other cells of the grid:

• Rewrite only — "does the retry budget setting still work with client library version 3" — moves the answer to rank 2. The topic is now correct and the text is still a question, so the FAQ still wins.
• HyDE only moves it to rank 3, and the top result becomes changelog#v3 at 0.985. HyDE generated a statement-shaped document about the only topic the raw query contained, which was "versions". It fixed the shape and amplified the underspecification.
• Both puts the answer first at 0.985.

That is the whole relationship between the two techniques, and it is worth stating as a rule: rewriting fixes what the query says; HyDE fixes what the query looks like. Neither touches the other's failure. A follow-up turn missing its referent cannot be repaired by making it more document-shaped, and a well-formed question does not get better by being restated — try the rewrite toggle on the 429 query and watch every number stay where it was.

The order matters too, and it is forced: rewrite, then generate. HyDE generates from whatever text you hand it, so handing it an unresolved pronoun means generating a confident paragraph about the wrong subject.

Rewriting also covers ground HyDE cannot. Ma and colleagues' Rewrite-Retrieve-Read frames it as closing the gap "between the input text and the needed knowledge in retrieval", and trains a small rewriter with feedback from the reader rather than assuming a prompt is enough. And the same generate-then-search idea works on the lexical side: Wang, Yang and Wei's query2doc appends a few-shot generated pseudo-document to the query and reports BM25 improving 3% to 15% on MS MARCO and TREC DL with no fine-tuning — which is a useful sanity check that the effect is about giving the retriever more of the answer's vocabulary, not about anything specific to embeddings.

Rewriting has its own failure and it is the mirror of HyDE's: a rewriter that normalises a query can delete the one token that mattered. "Why does ERR_CONN_2043 happen on deploy" becoming "why do connection errors happen during deployment" is a better sentence and a worse query. If you rewrite, log the before and after, and add a rule that any token matching an identifier pattern survives verbatim.

Checking it on a real system

Log the generated document. Every time. This is the single highest value thing in this lesson and almost nobody does it — there is an open issue against LlamaIndex because HyDE's generation was not appearing in traces at all. When retrieval goes wrong with HyDE in the pipeline, the question is never "why did the retriever do that"; it is "what did the model write". Put the generated text next to the retrieved chunks in your trace and most debugging becomes reading.

Measure the encoder gap before you build anything. Take fifty question-and-answer pairs from your own corpus. Embed each question, embed its answering chunk, and record the cosine. Then embed the answering chunk against a different, unrelated chunk and record that cosine. If question-to-answer similarity is systematically lower than chunk-to-chunk similarity for unrelated chunks, your encoder has the gap HyDE exploits and HyDE will help. If they are comparable, it will not. That experiment takes an hour and settles the question before you put a language model on the critical path.

Budget the latency honestly. HyDE adds a full generation before retrieval starts, and N generations if you follow the paper. It is serial: no part of the retrieval can begin until the text exists. In an interactive search box that is usually the end of the discussion. Mitigations that actually work are caching generated documents by normalised query — head queries repeat far more than people expect — capping the generation at roughly the length of one of your chunks, since a hypothetical document much longer or shorter than the things it is being compared against matches them worse, and using a small fast model for it, remembering that the paper measured smaller instruction models producing smaller gains.

Route rather than switch on. The four queries in the panel want different treatments: a conversational turn needs rewriting, keywords and well-formed questions benefit from HyDE, and a query containing an exact identifier or asking for a specific numeric default should go straight to the lexical retriever with the user's own text. A cheap classifier over query length, question-mark presence and identifier patterns captures most of that. HyDE's own authors end their paper describing routing: as a supervised retriever improves, "more queries will be routed to it, with only less common and emerging ones going to HyDE backend."

Evaluate at fixed k with the answer labelled, not by eyeballing. Report recall at your candidate depth for four configurations — plain, rewritten, HyDE, rewritten plus HyDE — and look at the per-query table rather than the mean. The signature of a HyDE regression is a small average improvement hiding a handful of queries that went from rank 1 to unranked, and those queries will nearly all be the identifier-shaped ones. Then run a cross-encoder over whatever HyDE returned, because the two stages compose: HyDE changes which candidates you have, the reranker changes their order, and only the first of those can raise recall.

One thing not to do. Do not feed the generated hypothetical document to the model as context. It is fabricated text that looks exactly like a retrieved chunk, it is the wrong shape of thing to cite, and it will be cited. It exists to produce a vector and it should be discarded the moment the search returns. What reaches the prompt is chunks, and every one of them costs prefill.