Everyone can read the filings. Almost nobody joins them.

If you read nothing else, read the six claims below. Each one is falsifiable, each one is defended later in the paper, and each one has a cost attached in section 13.

A useful way to decide what to build is to ask, for each layer of the stack, how long a competent competitor would need to reproduce it from scratch. Layers that take a fortnight are commodity no matter how much they cost us. Layers that take years are the business.

The estimates below are deliberately unkind to ourselves. They assume the competitor is good, has money, and knows what they are doing. What survives that test is worth funding.

This is an uncomfortable allocation, because the commodity layers are the ones that demo well. A new source lights up a dashboard. A better resolution threshold does not. The discipline is to keep funding the invisible half.

Start with a question a client would actually ask. Which UK local authority pension funds changed their emerging markets equity manager in the last two years, and what were they paying the outgoing one?

Every word of that question is answerable from public documents. None of it is answerable from any single public document. The answer lives across a committee report published as a PDF on a council website, an annual report filed months later, a regulatory return in a different format under a different identifier, and a manager’s own disclosure that names the mandate but not the client.

So the work is not retrieval. The work is assembly, and assembly across sources that were never designed to be assembled has four specific failure modes.

None of this is novel. The academic literature on truth discovery, the problem of deciding what is true when sources of unequal reliability disagree, has spent fifteen years on exactly this, and the finding is consistent: systems that model source reliability explicitly outperform majority voting, and systems that emit calibrated probabilities outperform systems that emit confident single answers. Google’s Knowledge Vault work is the best‑known industrial version of the same result.

The design in the rest of this paper is an attempt to take that finding seriously rather than rediscovering it in production.

Before designing anything, we catalogued the ground. 87 sources across the US federal system, US states, the UK, entity registries, market data and macroeconomic series. For each one we recorded what it actually contains, how it is obtained, what keys it carries, and the one thing about it that will bite an engineer who assumes it behaves normally.

78 were confirmed reachable at the terms described. 9 are recorded from documentation rather than fetched: eight because they are paid, gated, licensed or restricted in redistribution, and we did not buy past the terms, and one, the Australian holdings disclosure, free but assessed from secondary sources. 5 rows are marked as walls: 3 commercial, where access is sold or licensed rather than published, and 2, the PBGC (the US pension insurer) event filings and SEC Form PF, closed by statute, where no amount of effort or money helps. The walls are marked, not hidden, because a register that only lists what is easy is a sales document rather than a plan.

Open any row for the field notes. Filter by category or availability, or type to search across every field. Everything in this section is data, not prose: the counters above and in the opening are computed from these rows when the page loads.

Here is the map that makes the case better than any argument. Seventeen identifier systems, one institution. An identifier system is simply the numbering scheme a regulator or registry uses to name the entities it tracks; each was designed alone, for its own purpose, and none was designed to line up with the others. Solid lines are joins a machine can make from a published crosswalk, a table that translates one scheme into another. Dashed lines are joins that have to be inferred, defended, and occasionally corrected by a human.

Notice where the dashed lines cluster. The Legal Entity Identifier is the closest thing to a hub and it works well for banks, listed issuers and UK companies. It does almost nothing for the American private retirement world, which is precisely the part of the market an intelligence product in this space most needs to cover.

Architecture arguments are easy to wave away. Specific misbehaviour is not. Six examples from the register, each of which breaks a naive pipeline in a different way, and each of which the design in sections 07 to 11 exists to absorb.

The common thread is that none of these are bugs in the sources. They are the sources working as designed, under legal and administrative constraints that will not change because we find them inconvenient. A data layer either models them or is quietly wrong about them.

The first layer has exactly one job, and the discipline is in refusing to give it a second. Bronze stores what the source served, unaltered, addressed by the hash of its own content, with the retrieval circumstances recorded alongside it.

No cleaning. No renaming. No dropping of fields we do not currently use. The moment bronze starts improving things, the thing it exists to be, a faithful record of what was published, stops being true, and every downstream claim about provenance becomes an approximation.

This is the section where the architecture earns or loses its keep. Silver turns raw documents into typed records, and it does so on two independent time axes: when something was true in the world, and when we came to know it.

Those are different, they routinely disagree by months, and collapsing them is the single most common irreversible mistake in financial data engineering. Irreversible because the information is not lost in storage, it is never captured. A system that records only the current value cannot be upgraded later into one that remembers.

Concretely: a holding is reported as of the thirty‑first of March, published in mid May, amended the following February. Ask on the first of June what that position was, and the honest answer is the May figure, because the amendment did not exist yet. A single‑clock store answers with the February number and cannot tell you it has done so.

Gold answers two questions that silver deliberately leaves open. Which of these records describe the same institution, and where they disagree, which value do we publish and how sure are we.

Resolution, in three passes

Comparing every record against every other is quadratic and pointless, so the work is staged. Block to reduce candidate pairs using cheap similarity: trigram matching on names, which compares overlapping three‑letter fragments so spelling variants still match, and vector search on embedded descriptions, which compares meaning rather than spelling. Both of these the database can index. Score the surviving pairs with a probabilistic record linkage model that learns field weights from the data rather than from a hand‑written rule set. Adjudicate only the band in the middle, where the score is neither confidently a match nor confidently not.

Most pairs never reach the third pass, which is the point. The expensive judgement is spent where it changes an outcome.

Ranking, by earned authority

When two sources give different answers, the winner is decided by a tiered authority model: a primary regulatory filing outranks a regulatory summary, which outranks an entity registry, which outranks a market feed, which outranks a document a human wrote. The tiers are recorded, applied consistently, and revisable when evidence says they are wrong.

The loser is not deleted. It is published alongside, as a recorded disagreement, because a client who can see that two regulators differ is better served than one handed a clean number that hides it.

There is a version of this kind of product where analysts read filings and type the results into a system. It works, it is how most of this market was built, and it does not scale past the number of analysts you can afford.

There is another version where a model does everything and nobody checks. It scales beautifully and produces a product that cannot be sold to anyone with a compliance function.

The design here is neither. Machines carry the volume and are required to declare when they are unsure. Experts see only the cases that were escalated, and their decisions are captured as labelled examples that make the next round of automation better. The human cost per record falls over time while the quality per record rises, which is the only shape of this kind of business that compounds.

Everything above exists to make one thing possible: handing a client a figure that holds up when their own analyst goes looking for it.

That is a higher bar than accuracy. An accurate number with no provenance is worth very little to an institution, because the person receiving it cannot put their name on it. A number with the filing behind it, the retrieval time, the competing values and the reason this one won, is something a professional can defend in a meeting. That is the product.

The vocabulary for expressing this is a settled standard rather than something we need to invent: W3C PROV. Entity, activity, agent, and derivation relations between them are enough to state exactly where a value came from and what happened to it on the way.

A data layer is not a project that finishes. Sources change format without notice, endpoints move, regulators revise schemas, and a pipeline that was correct in March is quietly wrong in September. The operational design matters as much as the schema design.

Four commitments keep it honest, and all four are cheap if they are built in and expensive if they are retrofitted.

A paper that only argues one side is a pitch. Here is the strongest honest case against everything above, followed by what it would cost to be right.

Objection one: this is over‑engineered for an early‑stage product

Largely fair. A single table refreshed nightly would answer most questions a first customer asks, and would exist in six weeks rather than six months. The argument for the fuller design rests entirely on the belief that the point‑in‑time and provenance questions arrive and cannot be answered retroactively. If that belief is wrong, the simple version wins.

Objection two: breadth beats depth in a sales conversation

True, and it is the objection we take most seriously. Nobody has ever lost a bake‑off for having insufficiently rigorous lineage. People lose them for missing a region. The answer is not to argue the point but to sequence so that visible coverage and invisible rigour advance together, which is what section 14 does.

Objection three: the hardest data is the data we cannot get

Also true. Some of the most valuable disclosure in this market is closed by statute rather than by paywall, and no architecture opens it. What the design does is stop us spending repeatedly on things that will never work, by recording the closure once and marking it.

Objection four: bitemporal modelling is genuinely harder to work with

Yes. Queries are longer, mistakes are subtler, and someone new to it will write a wrong query confidently in their first week. Mitigations exist, chiefly a small set of reviewed views that handle the common access patterns, but the learning curve is real and should be budgeted.

This section is the wish list, in the order we would want it granted. Nothing in it asks for a commitment; each phase is shaped so a team could start small, produce a number, and stop without regret, with every later decision still open.

Nothing in this paper is invented here. The architecture is assembled from published, battle‑tested prior art, and the failure modes it defends against are documented in the open literature. Canonical links for the individual data sources live on their register rows in section 04.