Chunk Boundaries and the Size Trade

A fixed-size splitter cuts your documents at a character offset produced by arithmetic on their length. Nothing about that offset knows where an answer starts or stops. So whether a given question is answerable from your index is decided by where the division landed — and re-splitting the same document at 420 characters instead of 400 makes a different set of questions unanswerable.

There is no optimum to find. There is a lottery, and chunk_size buys a ticket.

The reason is not that half an answer is worse than a whole one, though it is. It is that the half holding the fact usually loses the words that made it findable. Technical prose refers back constantly — "it must be rotated", "its retry budget", "the city" — and the noun those words refer to sits in the previous sentence. Cut between them and you get one chunk that names the thing and does not answer the question, and one chunk that answers the question about a thing it never names. Neither matches the query. The document still contains the answer; your index no longer does.

The Late Chunking paper measured exactly this on Wikipedia. Against the query Berlin, the sentence "Its more than 3.85 million inhabitants make it the European Union's most populous city" scores a cosine similarity of 0.7084 when it is embedded as its own chunk, and 0.8249 when the same words are embedded with the rest of the article in scope. The text did not change. The word "Berlin" was in a different chunk.

Below is a runbook, a real splitter, and a real retriever. Leave everything alone at first and read the chunk-size lottery strip: each cell is one value of chunk_size from 80 to 800, and its colour says whether the current question is answerable at that size.

The strip is not a curve you could have optimised. With the fixed splitter, no overlap, and the first question, 11 of the 37 sizes fail outright, and they are not the 11 you would guess: 180 and 200 work, 220 through 320 all fail, and everything from 340 up works. Nothing in the document says that 200 is safe and 240 is not. What decides it is whether some multiple of chunk_size happens to land between offsets 212 and 322, which is where this answer lives — arithmetic on the document's length, meeting a span whose position nobody chose.

Switch the question to the second one and the pattern is starker, because its answer is a short self-contained sentence. It fails at 120, 140, 220, 260 — and then at 520, where 500 and 540 both work — and again at 780 between two working neighbours. A single unlucky size, surrounded on both sides by lucky ones. That is what a lottery looks like when you plot it.

Now set chunk_size to 280 and read the ranked list. The top result, at 0.651, is a chunk that begins: "It must be rotated within 30 days of issue, or the node begins rejecting inbound requests." It ranks first. It is about the right subject. It never says what It is. The sentence naming the signing certificate is in the chunk before it, and the model is now being asked to answer a question about certificates from a passage that does not contain the word. This is the failure that looks like a hallucination in the logs and is a splitter bug in the index.

Now switch the splitter to recursive. The first question goes from 26 working sizes to 33, and the failures collapse to the small end where a paragraph no longer fits in a chunk at all. That is the whole reason RecursiveCharacterTextSplitter is the default in every tutorial: it prefers to cut where the author already cut, and a paragraph break is a human's own statement about where one idea ends. It is not a smarter algorithm. It is the same algorithm reading a signal the fixed one throws away — and when chunk_size drops below the paragraph length, it has no signal left and cuts between two words like everything else.

Overlap is not the safety margin you think you bought

Go back to the fixed splitter, take the first question, and drag chunk_overlap from 0 to 200. Two things happen, and only one of them is the one you wanted.

The good one: red cells in the lottery turn green. An answer span of L characters is rescued by an overlap of o only when o ≥ L — the trailing copy of the previous chunk has to be long enough to hold the whole span. This answer is 110 characters long, and the strip says exactly that: at overlap 0 it works at 26 sizes, at 60 it works at 29, and at 110 it works at 35 of 37. The two that still fail are 80 and 100, where the overlap cannot be 110 because it cannot exceed the chunk. Overlap is a bound on the length of answer you are prepared to protect, and you are choosing that bound without knowing the distribution of your answer lengths.

The bad one: read characters stored. At chunk_size = 400 with an overlap of 200 the index holds 3,809 characters of a 2,009-character document — 1.90× the corpus, in vectors as well as in text. And push the overlap to 200 and the lottery gets worse, from 35 working sizes to 33. Set the size to 260 with overlap 200 to see why: the index is now 31 chunks and 3.99× the document, the chunk holding the answer scores 0.647 at rank 4, and one of the three shortlist slots is filled by a chunk that repeats more than half of another chunk already in the list. You asked for three pieces of evidence and received two. Overlap does not add information; past a point it just spends your context window on text the model has already read.

And often overlap does nothing at all. Switch to the recursive splitter, leave chunk_size at 400, and drag the overlap from 0 to 100 while watching the chunk count and the stored-characters readout. They do not move: 6 chunks, 1.00×, at every value. Only at 150 does anything happen, and then abruptly — 8 chunks and 1.27×. This is not a quirk of this simulation — it is LangChain's documented behaviour, reported as langchain#34804, "TextSplitter chunk_overlap is silently ignored unless chunk_size overflow occurs", and again as langchain#30200. Overlap lives inside the merge loop:

if total + len(next) + sep > chunk_size:
    emit the current chunk
    while total > chunk_overlap: pop from the front

If every paragraph already fits inside chunk_size, the splitter never enters that branch, so it never pops, so it never carries anything forward. The parameter is read and ignored. Your ingestion job logs no warning and your index is built exactly as if you had passed zero.

The trade has two ends and both of them are real

Put chunk_overlap back to 0 and the splitter back to fixed — the previous section left both somewhere else, and every number here is at zero overlap. Then set the question to the third one, "which component issues the signing certificate?", and walk chunk_size down from 800 while watching the score of the chunk that holds the answer.

It climbs the whole way: 0.524 at 800, 0.568 at 400, 0.662 at 300, 0.722 at 200. Same document, same answer, same query, same encoder. The only thing that changed is how much other material is averaged into the vector that represents the answer. At 800 the runbook is three chunks, and the one holding the answer is also holding paragraphs about token expiry and log shipping; its direction is the mean of all of them, so it points a little at each and squarely at none. This is the pooling geometry doing exactly what it always does, and the splitter is what decides how much of it you suffer.

Keep going down and the other end arrives. At 120 the strip turns red for this question: the 44-character answer no longer survives a stride of 120 with a cut landing where it does, and no chunk contains it. You have built an index of fragments that rank beautifully — the top score at 100 is 0.940 — and cannot be quoted.

This is the trade, and it is not resolvable by choosing better:

• Bigger chunks raise the chance the answer is in there and lower the chance it ranks. Containment goes up, and every extra sentence pulls the pooled vector further from the sentence you needed — the mean pooling geometry is the mechanism, and it does not care that one of the sentences is the important one.
• Smaller chunks raise the chance it ranks and lower the chance it is complete. And they multiply the number of vectors: the approximate index is now walking a larger graph at the same ef_search, so its own recall drops at the same time.

The way out is not a number. It is to stop making the indexed unit and the returned unit the same object. Index a small, precise chunk so that it ranks; store a pointer to the surrounding parent block; return the parent to the model so the answer is complete. That is what "small-to-big" and LlamaIndex's parent-document retrieval do, and it works because it declines the trade rather than optimising it. The cost is that everything downstream now handles two units: your citation offsets, your reranker, which scores the text it is given and cannot re-join what the splitter separated, and your context budget, since the parent block is what gets prefilled into the KV cache.

Checking it on your own corpus

The measurement that matters is not "what is the best chunk size". It is "which of my questions are currently unanswerable", and it takes an afternoon.

1. Take 30 real questions. For each, open the source document and copy the exact span of text that answers it — the sentence or two a human would point at. Store the document id and the character offsets. This is the only labour in the whole exercise, and it is the asset: it survives every change to your pipeline.
2. Run your ingestion job. For each question, ask one boolean: does any single chunk contain the whole span? Not "is it similar", not "does it retrieve" — does the text fit inside one chunk. Call the fraction that do your containment rate. If it is 0.7, then 30% of your questions cannot be answered correctly by any retriever, any reranker, or any model, and no amount of prompt work will move them.
3. Only for the questions that pass containment, measure whether the containing chunk is retrieved at your top_k. That is recall@k, and it is a fair number only when it is conditioned on containment. Reported unconditionally it silently mixes a splitter bug with a ranking bug.
4. Re-run steps 2 and 3 for four or five chunk sizes. You are not looking for the maximum — the strip above shows what that surface looks like. You are looking for whether the sizes disagree about which questions fail. If they do, and they will, that tells you the failures are boundary accidents rather than a size problem, and the fix is structural.

Two concrete things to check in your existing pipeline today. First, print the actual chunks — print(repr(chunks[i])), not the count. If they start mid-word, your separator list never reached a boundary that existed in the document. Second, verify that your overlap is real: chunks[i][-overlap:] == chunks[i+1][:overlap] should hold for some i, and on a document of short paragraphs with the recursive splitter it will hold for none of them.

And before tuning anything, check the boring failure. The encoder has a hard input limit that is usually shorter than the chunk you are feeding it: all-MiniLM-L6-v2 ships max_seq_length: 256 word pieces. A 1,000-character chunk is roughly 250 tokens, which just fits; a 2,000-character chunk does not, and the second half of every one of those chunks was silently truncated before it was ever embedded. That is not a chunking trade-off. That is text you believe is in your index and is not.