Data Lineage Tracking
Concept
Data lineage is the record of where a piece of data came from and every transformation it passed through on the way to where it's being read now — which source table fed a pipeline, which job transformed it, which intermediate dataset it landed in, and which downstream dashboard or report ultimately consumed it. The problem it solves is diagnostic: once data flows through more than one stage (raw ingestion → warehouse → transformation job → dashboard), a wrong number in a report is no longer something you can eyeball — it could have been introduced at any of several hops, by a schema change, a bad join, a stale partition, or a silently failed upstream job. Without lineage, tracing it back means manually reading pipeline code and asking every team along the chain "did anything change recently"; with it, you query the lineage graph and get the actual dependency path.
Lineage has historically been maintained two ways, with the industry converging hard on one of them. Manually-maintained documentation (a wiki page describing "the pipeline") rots the moment a job changes and nobody updates the doc — it's actively misleading past a certain team size. The alternative that has become the de facto standard is automatic, run-time-collected lineage: instrumenting the tools that actually move data (Spark, Airflow, dbt) so that every job run emits structured facts about its inputs, outputs, and the run itself, without a human transcribing anything. OpenLineage, an open specification introduced in 2020, is the standard this converged around — it defines a common model of Job, Run, and Dataset entities with consistent naming, so that Airflow, Spark, and dbt can each describe "this run consumed dataset X and produced dataset Y" in a shared format regardless of which tool did the work. A collector (e.g., Marquez, or a cloud data-catalog product) ingests these events from every tool in a mixed stack and assembles them into one queryable graph, rather than each tool maintaining its own disconnected notion of lineage.
Tradeoffs
| Approach | Accuracy | Maintenance cost | Coverage |
|---|---|---|---|
| Manual documentation (wiki, diagram) | Degrades immediately as pipelines change; only as good as the last person who remembered to update it | Low to build, but requires ongoing discipline nobody consistently applies | Whatever someone thought to document — invisible for anything unwritten |
| Column-level static analysis (parsing SQL/job code without running it) | Reasonably accurate for lineage that's fully expressed in the parsed language | Moderate — needs a parser per language/tool, breaks on dynamic SQL or non-SQL transforms | Misses lineage introduced by code the parser can't see through (a Python UDF, a dynamically built query) |
| Automatic runtime collection (OpenLineage-style event emission) | Reflects exactly what actually ran, including dynamic behavior, because it's captured from the real execution | Moderate upfront (instrumenting each tool/job), near-zero ongoing since it's automatic thereafter | As complete as your instrumentation coverage — a pipeline stage that isn't instrumented is a gap in the graph, not a wrong answer |
Runtime collection wins on accuracy and long-term cost, but it isn't free: it only covers what's instrumented, so a stack with one instrumented orchestrator and several uninstrumented ad-hoc scripts still has real blind spots — the lineage graph will look complete and simply be silently missing a branch.
When to use / when not to
- Use lineage tracking once data passes through more than one transformation stage feeding a shared destination (a warehouse, a BI dashboard) — a single raw-to-destination pipeline with no intermediate stages doesn't have much to trace.
- Prioritize it wherever a wrong number has real consequences — financial reporting, compliance-driven data (see Data Retention and Deletion Policy for the adjacent "how long does this data live" question), or any dashboard executives make decisions from.
- Don't retrofit full runtime lineage collection onto a one-off batch script that runs twice and gets deleted — the instrumentation cost isn't worth it for throwaway pipelines.
- Column-level static analysis is a reasonable lighter-weight starting point when the entire stack is expressed in one well-parseable language (e.g., all dbt/SQL) and dynamic, code-generated queries are rare.
Common pitfall
Treating a lineage graph as complete once it exists, without auditing instrumentation coverage. A graph that's genuinely missing a branch (an uninstrumented legacy script feeding into an otherwise fully-instrumented warehouse) doesn't present as "incomplete" to whoever's using it during an incident — it presents as a dead end that looks like a source, sending the investigation to the wrong root cause. The practical fix is to track instrumentation coverage as its own metric (what fraction of known pipeline stages emit lineage events) rather than trusting that the graph shows everything just because it shows something.
Engineering Lens
Lineage is really an observability concern applied to data rather than to service calls — the same instinct that produces distributed tracing for a request crossing microservices produces lineage for a value crossing pipeline stages, and it should be evaluated the same way: not "do we have a lineage tool" but "if this number is wrong tomorrow, how many hops can we actually trace before hitting an instrumentation gap." Adopting a shared standard like OpenLineage rather than a vendor-specific format matters for the same reason a shared tracing context matters in distributed tracing — it's what lets tools from different vendors (an orchestrator here, a warehouse there) contribute to one coherent graph instead of three disconnected ones that each need separate reconciliation during an incident.