Extracting Requirements from Long PDFs with LLMs: What Silently Breaks and What Actually Works

I run a small company that reads public tender documents for a living. Over the past year we processed tens of thousands of them, and for a published study we analyzed 68,717 notices across the UK, EU, US and Australia. A typical package is a few hundred pages across a dozen files, and the job is unforgiving: extract every binding requirement, each with a page citation, because one missed "mandatory certification, see page 74" disqualifies a bid.

That makes it a good stress test for LLM document extraction. Not summarization, where dropping detail is the point, but exhaustive extraction, where dropping detail is the failure mode. Here is what broke for us in practice, mostly silently, and what we do about it now.

1. Long documents fail in the middle, and the output looks fine

The nastiest property of extraction failures is that the output is well-formed. The model returns a clean list of requirements with confident page references, and nothing tells you it stopped reading.

Our worst regression: a 99-page NHS contract we use as a benchmark went from 366 extracted constraints to 184 in one week. No errors anywhere. When we diffed the results by page, the extraction covered roughly the first 25 pages and a thin sample after that. The model had not refused or crashed. It had quietly under-read, and the missing 180 constraints included an annex holding the evaluator-critical thresholds.

The fix that held up was boring: chunk by explicit page ranges, extract each chunk independently, and union the results. But chunking introduced its own silent failure. When one chunk returns an empty response (a transient API error, a safety filter, anything), a naive union just absorbs it. An empty chunk looks exactly like a chunk with no requirements. We now treat "chunk returned zero items" as an anomaly to retry and log, not a result, after losing about 20 pages of that same NHS document to exactly this.

The general lesson: for exhaustive extraction, coverage must be measured, never assumed. We track constraints-per-page-region on benchmark documents, and a distribution change is treated as a regression even when the total count looks plausible.

2. Page citations are harder than extraction

We require every extracted requirement to carry a page-level citation back to the source. That single constraint surfaced three separate bugs that pure-text extraction would never have shown us.

First, printed page numbers are not PDF page numbers. Engineering-contract documents (FIDIC-style in our case) print their own page numbering per section, and the model happily cites the printed number while the viewer opens the PDF index. We had phantom citations that were off by an entire front-matter's length, and for a while one code path even added the two numbers together.

Second, annexes restart numbering. "Page 5" can exist four times in one package. Citations have to be qualified by document part, or they are ambiguous precisely where it matters most, since annexes are where the thresholds live.

Third, format drift. Two code paths emitted "p. 5" and "5", and downstream verification treated them as different pages. Trivial, embarrassing, and invisible until we audited citation accuracy end to end.

The payoff for this pain is a hard quality gate that has no equivalent in citation-free pipelines: a verifier can open the cited page and check whether the requirement is actually there. When we first measured our verifier honestly, it was under-reporting badly, flagging 68 of 158 claims as unverified when a manual audit showed essentially all of them were grounded. The verifier was string-matching too literally. Verification code needs the same skepticism as extraction code.

3. Run-to-run variance is larger than most teams expect

Same document, same prompt, same model, temperature already low: our extraction counts varied from 35 to 61 constraints across runs on one benchmark, a 75 percent spread. If you run extraction once and ship the output, you are sampling a distribution and pretending it is a value.

We moved to best-of-N with a twist that mattered: the winner is selected by a quality rubric, not by count. Picking the run with the most extracted items rewards over-extraction, and over-extraction is a real failure mode too; duplicated near-identical requirements inflate counts while adding review burden. Deduplication needs to catch reworded siblings, not just exact matches, which is itself a small semantic-similarity problem.

Cost note: N passes do not have to mean N uploads. Uploading the document once and sharing one cached context across all passes cut the marginal cost of best-of-N substantially. Context caching is the single most underused feature in this kind of pipeline.

4. Tables need a different path

Pricing schedules and requirement matrices arrive as spreadsheets or as dense PDF tables. Text-first extraction mangles them. We route tables through a vision pass with generous timeouts (our table-extraction timeouts are minutes, not seconds), and we watch for the batch-level version of the empty-chunk problem: one of five table batches returning zero rows is usually a silent failure, not an empty table.

5. The evaluation harness is the product

None of the above was found by a demo. All of it was found by a fixed benchmark corpus with known-good annotations, re-run on every meaningful change. Ours is small: a handful of real documents, a few hundred hand-verified requirements, per-page coverage stats, and a rubric score. It is the only reason we caught the 366-to-184 collapse as a regression rather than shipping it.

If you are building extraction over long documents, my honest ordering of effort: benchmark corpus first, chunking with coverage accounting second, citation discipline third, best-of-N selection fourth, prompt wording last. We spent our first months in the reverse order, and paid for it.

What I would tell a team starting today

• Treat well-formed output as untrusted. Measure coverage by page region, not by total count.
• Make citations first-class and verify them mechanically. A finding without a checkable citation is a rumor.
• Assume run-to-run variance until you have measured it. Best-of-N with quality-based selection beats single-shot everywhere we tested.
• Log empty responses at every granularity (document, chunk, table batch) as anomalies, never as results.
• Long-context models raised the ceiling, but they did not remove any of these failure modes. They just made the silent versions cheaper to produce at scale.

The reading problem is genuinely solvable now, which was not true three years ago. But the gap between "solvable" and "solved in production" is exactly the list above, and every item on it failed silently first.

Davor Jerković is the founder of Lucius AI. The pipeline described here powers our tender analysis, and the 68,717-notice dataset behind the study figures is described in The State of Public Procurement 2026.