What is data provenance?

Data provenance is the recorded history of a piece of data: where it originated, what transformed it on the way, and who or what touched it since. The word comes from the art trade, where a painting's provenance is the chain of ownership that makes it worth something — and the analogy holds, because in both cases the object without its history is worth dramatically less.

In data engineering the term is often used interchangeably with lineage. There is a useful distinction: lineage usually describes relationships between datasets and jobs (this table is built from those two by that pipeline), while provenance is normally finer-grained — the history of an individual value.

Why AI changed the stakes

Provenance used to be a compliance concern and a debugging aid. It is now the main defence against a specific failure that has no other defence.

A language model produces fluent text whether or not it has evidence. An answer assembled from four cited records and an answer invented from priors are typographically identical. There is no tell, no hedge, no formatting difference. The reader has no way to distinguish them, and so does not try.

Provenance is what makes the difference visible. Not because a citation reassures anyone — most are never clicked — but because it makes a wrong answer findable. An uncited figure cannot be audited, so in practice it never is. A cited figure that points at the wrong record is obviously wrong to anyone who looks, which is a far lower bar than truthfulness and a much more achievable one.

That is the honest claim to make about provenance: it does not make a system correct. It makes a system checkable.

What a usable citation contains

Plenty of products display something citation-shaped that cannot actually be verified. The difference is whether a reader can get from the claim to the evidence:

• The record, not the system. "According to your CRM" is not provenance. A link to contact 0012f, field renewal_date, is.
• A timestamp — when that value was read, which matters the moment the source changes.
• The transformation, if any. A figure that was summed, converted between currencies or rounded should say so, because the arithmetic is where disputes land.
• Stability. A citation that rots when the source is edited was documentation, not evidence. This is where provenance meets bitemporal storage: if a correction supersedes rather than overwrites, the original citation still resolves to what it originally said.
• Permission-awareness. A citation must not be a side channel. Showing a reader the existence of a record they cannot open leaks through the footnote.

Where it is required rather than nice

• Regulated reporting, where a figure has to be traceable to a source document.
• The EU AI Act and comparable regimes, which lean on record-keeping and traceability for higher-risk uses — the practical reading of which is that you must be able to reconstruct how an output was produced.
• GDPR erasure. You cannot honour a deletion request across derived data unless you know what was derived from what. Erasure that propagates is a provenance feature before it is a privacy feature.
• Incident review. When an automated action went wrong, the first question is always what evidence it acted on.

The gap most systems have

A great many pipelines record provenance at the dataset level and nowhere near the value level. You can establish that a table was built from a job that read three sources; you cannot establish which of those three produced the number in row 400,000, column 7. For analytics that is usually tolerable. For an assistant quoting a single figure to a customer it is not, because the unit of dispute is the figure.

Retrofitting is painful for the obvious reason: provenance has to be captured at write time by every path that writes, and any path that does not will quietly produce facts with no history — indistinguishable from the rest until somebody asks about one.

Memtro records provenance per fact rather than per dataset: every figure cites the record it came from, corrections supersede rather than overwrite, and because the graph is bitemporal the citation still resolves to what it said when it was made.