Lineage-Driven Incident Response for Data Pipeline Failures

A freshness check failed at 6:12 AM. The on-call engineer opens Airflow — all green. The warehouse has rows. The dashboard is still wrong.

Without lineage and structured context, incident response becomes Slack archaeology: grep logs, ask who changed what, rebuild the graph from memory. Lineage-driven response starts from known dependencies and check semantics.

The incident timeline you want

1. Detect — check fails (not task exception)
2. Scope — lineage lists downstream dashboards and tables at risk
3. Diagnose — upstream tables and likely causes ranked
4. Verify — runbook steps (row counts, partition filters, load logs)
5. Communicate — stakeholders know which reports are affected
6. Document — run ID + check result + resolution linked in catalog history

DataXPipe supports steps 1–4 in the product today; 5–6 build on run and check APIs.

Blast radius from lineage

Register lineage in your pipeline spec:

lineage:
  edges:
    - from: mart_orders_daily
      to: mart_executive_kpis
      via: rollup_sql

When mart_executive_kpis fails freshness, query downstream and upstream edges. Typical questions:

• Which dashboards depend on this mart?
• Which upstream table stopped updating first?
• Did a deploy change this pipeline or an ancestor?

Failure explanation workflow

The Get started wizard demonstrates end-to-end:

1. Load stale-table demo catalog (or your registered pipeline)
2. Review generated checks and lineage
3. Simulate failure & explain — structured summary, blast radius, recommended actions

In production, the explain API accepts check failure payloads and lineage context to return:

• Human-readable summary (or enhanced summary when available)
• Likely causes (empty extract, partition filter, late upstream)
• Recommended actions (verify row counts, check upstream freshness)
• Blast radius lists (upstream nodes, dashboards at risk)

This is not a replacement for digging into logs — it is a prioritized checklist so you skip dead ends.

Pair lineage with check severity

Severity	Response
error on executive mart	Page on-call, comms to stakeholders
warn on secondary report	Ticket, fix in business hours
repeated warn on same target	Promote to error or fix root cause

Lineage tells you who to notify downstream; severity tells you how fast.

Runbook template

## Check: freshness_executive_kpis failed

**Run ID:** {from app}
**Observed delay:** 38h (threshold 24h)

### Upstream (lineage)
- [ ] mart_orders_daily — freshness OK?
- [ ] raw_orders load — row count vs yesterday?

### Downstream impact
- executive_kpis_dashboard (Looker)
- finance_daily_export

### Likely causes
- Zero-row extract
- Transform filter excluded latest partition

### Resolution
- {fill after fix}

Store runbooks linked to check IDs in your wiki; DataXPipe catalog holds the live graph.

Reduce repeat incidents

After resolution:

• Add volume check if zero-row loads are common
• Add not-null on grain columns if transforms produce empty grains
• Fix ownership in pipeline spec if the wrong team was paged
• Register schema changes only after lineage impact review

Try the demo path

No warehouse required: Get started → Try the demo → simulate failure → review explanation → register pipeline.

Related reading

• Silent failures when orchestrator is green
• Pipeline lineage best practices
• Incident response for data pipelines
• DataXPipe use cases guide

Lineage plus checks turns “the dashboard looks off” into a reproducible workflow — start with one critical mart and expand.