Our product reads public tenders, so the unglamorous half of the company is a set of pipelines that pull notices from ten official sources across seven countries, every day, without lying about what they captured. Over 68,000 tenders have come through them. This post is about the plumbing, because every platform in this market claims to "track everything" and almost nobody writes down what that actually costs.
It is a companion to the post about making the AI layer honest. Same thesis, one layer down: most of data honesty is not intelligence, it is maintenance.
Every source fails differently
SAM.gov (US) is the biggest firehose and the moodiest. It has broken on us with silent 401s after a mandated API key rotation, and then with 429 rate limits that only went away after we discovered the page-size sweet spot. Tens of thousands of its notices arrive thin, a title and little else, so there is a whole second refetch tier just to fill in what the first pass legally could not get.
The UK sources taught us about certificate pinning the hard way: when Let's Encrypt rotated toward its Gen-Y roots, our UK ingestion path needed an explicit trust update or it would have gone dark mid-cycle. Find a Tender also rate-limits unpredictably, so the fetcher runs a ratchet that backs off and recovers on its own. And our own audit once showed we were capturing about 12 UK notices a day when 100 to 200 were being published. Nothing was erroring. The queries were just narrower than the market. If you do not routinely audit capture against the source's own published volume, you are running on vibes.
TED (EU) publishes in all 24 official EU languages, and the majority of notices from smaller markets never appear in English at all. The pipeline ingests multilingually and analyses packs in their source language, which is less an AI achievement than a decision most competitors quietly avoid because it makes everything downstream harder.
Watermarks freeze silently. Incremental ingestion runs on last-seen-published-at watermarks, and our nastiest incident was a watermark that stopped advancing with no error anywhere. The tell, absurdly, was timeouts landing exactly at minute :59. A frozen watermark does not look like an outage. It looks like a quiet news day, for weeks, unless something is checking.
The failure map
The same content as a table, because this is the figure I wish someone had published before we started.
| Layer | How it failed | What runs now |
|---|---|---|
| SAM.gov | Silent 401s after key rotation; 429 storms; thin notices | Key-rotation alarms, tuned page size, a dedicated refetch tier |
| UK (CF / FTS) | TLS root rotation; erratic rate limits; quiet under-capture | Pinned trust updated for LE Gen-Y roots, self-recovering backoff ratchet, capture audited against published volume |
| TED | Majority of notices never exist in English | Multilingual ingest across all 24 EU languages, source-language analysis |
| Incremental sync | Watermark froze silently; weeks of "quiet news days" | Stall heuristics that treat a non-advancing watermark as an incident |
| Documents | Null bytes in PDF text layers poisoning JSONB writes | Byte-level sanitisation at the storage boundary |
| Taxonomies | NAICS/PSC to CPV mapping filed ~65% of US notices as "medical" | Rebuilt crosswalk, distribution monitored per country |
| Compute | Deploys killed in-flight analyses; recovery sweeper self-cancelled | Dedicated Cloud Tasks queues, atomic recovery claims, 167 ms pickup |
The data lies politely
Cross-source duplication is constant: the same contract appears on Contracts Finder and Find a Tender with different identifiers, so rows are fingerprinted on title, authority, and deadline before insert, and user uploads are deduplicated by SHA-256 with a freshness rule, because matching a new upload to a stuck failed row from last month is worse than reprocessing.
Two favourites from the long tail. PDF text layers sometimes contain literal null bytes, which Postgres JSONB rejects, so one malformed source document could poison a write until we stripped them at the boundary. And classification systems disagree: mapping US NAICS and PSC codes onto the European CPV taxonomy naively had filed roughly 65% of American notices under "medical". Every classifier downstream of that was confidently wrong about an entire country until we rebuilt the mapping.
The deploy that killed live analyses
The worst infrastructure bug we have shipped had nothing to do with data. Analyses used to run as in-process background tasks on Cloud Run. Every backend deploy drained the old instances, and any analysis in flight died with them. A real user uploaded a genuine hospital tender, our deploy landed minutes later, his analysis was killed, and the recovery sweeper we had built for exactly this case turned out to self-defeat: it enqueued the retry task before bumping the row's timestamp, the queue delivered in about 50 milliseconds, and the worker's freshness guard saw a just-touched row and skipped it. Every recovery attempt cancelled itself. The user asked to delete his account, and I do not blame him.
The rebuild moved all analysis work onto dedicated Cloud Tasks queues, with the pipeline running inside the request handler so a killed delivery is simply redelivered, and the recovery claim written atomically in the same statement that touches the row. Measured pickup latency on the dedicated queue: 167 milliseconds from enqueue to worker start. The same audit removed every LLM call that still ran inside a user-facing request, because a 120-second gateway timeout does not care how good your model is.
The retrieval layer
Everything lands in AlloyDB for PostgreSQL, which gives us plain SQL and vector search in one engine: ScaNN indexes over tender embeddings, hybrid retrieval that combines structured filters with semantic ranking, and IAM database auth so there are no database passwords to leak. Running the corpus and the vectors in the same database sounds like a detail until you have watched a two-store architecture drift out of sync. We have more to say about this layer soon, with numbers, in a piece we are finishing with people who know it far better than we do.
What I would tell anyone building on government data
- Audit capture against the source's own published volume. Silence is not health.
- Treat watermarks, dedup keys, and taxonomy mappings as production code with tests, not as scripts. Each one has cost us more than any model regression.
- Never run LLM work inside a request, and never let a deploy be able to kill customer work. Queues are boring. Boring is the feature.
- When a pipeline says "completed", make it prove coverage. The politely partial success is the failure mode that reaches customers.
I am Davor, founder of Lucius. The corpus these pipelines feed is public on our research page, and if you run ingestion against any of these sources and have scars I should know about: contact@ailucius.com.
Get help with your bid