Nobody Measures the Decision That Caps Everything Else
Everyone argues about the embedding model. Everyone tunes the reranker. Almost nobody measures the decision that quietly sets the ceiling on both: how you chop the document into chunks before any of that happens.
Cut on a fixed character count and you will slice clean through the middle of the sentence that held the answer, no ceremony, no warning. The retriever finds a chunk that's "near" the answer, the model reads half a fact, and it confidently makes up the other half like it meant to say that. The chunk boundary broke you, not the embedding, and nobody's putting that in the incident report.
Measuring the Only Thing That Matters
chunklab puts the common strategies side by side on a retrieval task where the answer spans are known exactly, and checks the only question that matters: did the retrieved chunk actually contain the whole answer, or just land near it.
• FixedSizeChunker: cuts every N characters. Fast, and it does not care what it slices through.
• OverlapChunker: the same idea with overlap between chunks, a cheap hedge against boundary casualties.
• RecursiveChunker: splits on paragraphs, then sentences, then a hard cut. Respects the document's own seams.
• SentenceChunker: never splits a sentence, so it never splits a one-sentence answer.
The Number
On a 10-question benchmark with a fixed retriever so chunking is the only variable, fixed-size cutting retrieves the whole answer 50% of the time. Overlap buys back a little at 60%. The structure-aware strategies, recursive and sentence, both hit 80%, a full 30 points free, sitting in a decision most pipelines make without ever measuring it.
I Actually Ran the "Swap in Your Own Documents" Test
The README already carried an honest caveat: this is one document, ten planted answers, a lexical retriever, not a law of chunking. That caveat sat there unverified until I actually did what it invites. I wrote a second document, a different domain entirely, a REST API reference instead of an internal wiki, different sentence lengths, different structure, with questions phrased before I ever ran the benchmark against it, and reran the identical measurement.
The 30-point margin nearly disappeared. Fixed-size cutting trailed the best structure-aware strategy by 10 to 20 points on the second document, not 30, and recursive, the strategy that tied for first on the bundled document, dropped to worst.
Digging into why turned up a real bug: the retriever's scorer has no stopword list at all, so "a," "how," and "does" count as matches with exactly the same weight as a genuine content word. On the bundled document this bug is completely inert, adding stopword filtering changes zero of the benchmark's outcomes there, which is precisely why nobody had noticed it. On the second document, the same one-line fix flips which strategy wins outright: recursive goes from worst to best. The bug didn't get worse. It was always there. It just happened to be silent on the one document the headline number was measured against.
The original chunkers and benchmark are untouched, so the bundled-document numbers above still reproduce exactly. The second document, the stopword-filtered retriever, and the side-by-side comparison ship as an explicit opt-in module, and the honest conclusion is right there in the output: the effect is real, the exact number is not universal, and now there's a harness that actually proves that instead of just saying it.
18 tests pass in CI across Python 3.9, 3.11, and 3.13.