This week we ran a live tender through our pipeline: the design and build of a 3G football pitch for a parish council in Gloucestershire, £950,000, six published documents, 133 pages. The AI produced a first draft in about five minutes. The draft opens like this:
Read this first: 11 requirements in this draft are answered by a delivery partner you have not named yet. Minimum turnover evidenced by 2 years of accounts, Professional Indemnity Insurance of £5,000,000, full roll carpet replacement without unauthorised patching, and others.
That banner is the feature I am proudest of, and it is a refusal. The model read the pack, understood what the buyer demands, checked it against what the bidder can actually evidence, and declined to write fiction for the gap.
Getting an LLM to do that reliably took us about a year of failures. This post is the postmortems. I run Lucius, an AI that reads public tender packs and produces compliance matrices and first drafts, but nothing below is a pitch. It is the specific ways a document AI lies to you, found the hard way, and what we changed each time.
Why fabrication is the default, not the edge case
A tender response is not marketing copy. When a supplier writes "we hold £5m professional indemnity insurance" in a bid, that is a representation the buyer will verify and can act on. Getting it wrong does not lower your score. It gets your bid set aside, and in the worst case it gets you excluded from future work.
A language model, left alone, optimises for the most plausible completion. On a bid document, the most plausible completion of "our certifications include" is a list of certifications, whether or not the bidder holds them. Plausibility and truth point in the same direction almost everywhere in the training data. In bidding they diverge exactly where the money is: on the claims the evaluator checks.
So the interesting engineering problem is not making the model write well. Current models write well by default. It is making the system know, row by row, what it is entitled to claim.
Postmortem 1: the phantom partner
In late July we audited a full draft for an NHS tender we use as a test
fixture. The draft read confidently. It also deferred 42 requirement rows to
a consortium partner, woven through the prose as if a partner arrangement
existed. There was no partner. The model had invented a load-bearing company
and named it [PARTNER_NAME], disclosed at the bottom of the
document, long after the prose had presented the arrangement as fact.
Worse, our own compliance verifier scored most of those 42 rows as covered. It matched keywords, and the phantom partner's paragraphs contained the right keywords. Two components, each roughly reasonable in isolation, composed into a system that wrote fiction and then certified it.
The fixes were structural, not prompt tweaks. Drafting now runs a capability-fit check against the bidder's actual profile before any section is written. Requirements the bidder cannot evidence are routed to an explicit, visible partner slot, and the count is surfaced in a banner at the top of the draft, not a footnote at the bottom. The verifier was rebuilt to stop crediting partner-framed paragraphs as the bidder's own evidence. And the system refuses to draft at all when its own bid recommendation says the tender is not worth bidding, because a beautifully written draft for a doomed bid is its own kind of lie.
Postmortem 2: the silent coverage collapse
The scariest failures in LLM pipelines are not crashes. They are partial successes that report as complete.
Our extraction stage once dropped from 366 extracted requirements to 184 on the same 99-page NHS pack, seven days apart, with every run reporting "completed". The cause was a chunked-extraction path that collapsed past page 33: one chunk returned an empty result, and the code quietly unioned an empty set into the total. Nothing errored. The pipeline was proud of itself. The annex containing the evaluator's threshold criteria was simply absent from the analysis.
If your system loses 50% of the requirements and tells the user "analysis complete", every downstream honesty mechanism is decoration. The fix was boring and essential: chunk-level results are no longer allowed to be silently empty, coverage is measured against page span rather than assumed, and a missed-items pass hunts specifically for what the first pass skipped. Boring is the point. Most of honest AI is plumbing.
Postmortem 3: the broken truth-meter
We verify citations: every extracted requirement carries a page reference and a quote, and a checker confirms the quote exists in the source. At one point that checker flagged 90 of 158 citations as unverified. Alarming, so we hand-checked a sample: 27 of 27 were actually grounded, quote and page correct. The instrument was broken, not the pipeline. It was failing on whitespace and hyphenation differences between the PDF text layer and the model's quote.
This one changed how I think about the whole category. If your truth-measuring instrument is wrong, you do not know whether anything else works, and both failure directions are poisonous: false alarms teach the team to ignore the meter, false passes teach the customer to trust fiction. We now treat the verifier as a measured component with its own eval set, the same way the extraction and drafting stages have theirs. Rulers before measurements.
What the system does today
The result of those postmortems, in one list. None of this is exotic. All of it exists because a specific failure demanded it.
- Every requirement is extracted with a page citation and a verbatim quote, and the quote is machine-verified against the source text.
- Extraction coverage is measured against the document span, and a dedicated second pass hunts for missed items. "Completed" is a claim the pipeline has to earn.
- Drafting is gated on the bid verdict. If the analysis says do not bid, the system says so instead of writing.
- A capability-fit check runs before drafting. Requirements the bidder cannot evidence are routed to a visible partner slot and counted in a banner at the top of the draft.
- Names the system does not know stay as bracketed slots. It will not invent a project manager.
The receipts
Because a post like this is cheap without artifacts: the run I opened with is published in full, unedited. The analysis found 45 mandatory requirements in the 133-page pack, 13 of them high severity, and 9 of those 13 are legal clauses sitting in the conditions of contract rather than the specification. The draft addressed 40 of the 45 directly and flagged the other 11 out loud. The full case study is on our research page and the actual PDFs are downloadable, honesty banner and all, with a clearly labeled fictional demonstration bidder because inventing a real one would rather miss the point of this post.
The opinion part
I think document AI vendors have the incentive structure backwards. The demo that wins the deal is the confident one, so the market optimises for confidence, and the cost of fiction lands later, on the customer, in a procurement context where fiction is disqualifying. The vendors will call their coverage numbers "hallucination-free". Ask instead: show me where your system says it does not know. If it never does, it is lying to you somewhere, and neither of you knows where.
In this category, the refusal is the product.
I am Davor, founder of Lucius. If you have shipped grounded document AI and disagree with any of this, I genuinely want the argument: contact@ailucius.com.
Get help with your bid