Catalog API Quick Reference

The DataXPipe Catalog is the system of record for pipeline metadata, run history, check results, and dataset lineage. Generated Airflow DAGs, check scripts, and the product UI all communicate with the same REST surface. This reference covers the endpoints you will use most often in production.

Base URL and authentication

All requests target your Catalog host (for example https://api.dataxpipe.com). The API is versioned under /api/v1/.

Authenticate with one of:

Method	Header	Use case
API key	`X-API-KEY: dxp_...`	CI/CD, generated DAGs, org-scoped keys
Bearer JWT	`Authorization: Bearer <token>`	Product UI, OIDC-integrated services

Legacy scaffold keys (platform-key, admin-key, dev-key) work in local development. Production deployments should use organization API keys or OIDC.

Pipelines

# Register pipeline metadata
curl -X POST https://api.dataxpipe.com/api/v1/pipelines/ `
  -H "X-API-KEY: dxp_your_key" `
  -H "Content-Type: application/json" `
  -d @generated/orders_sync/metadata/pipeline.json

# List pipelines
curl https://api.dataxpipe.com/api/v1/pipelines/ `
  -H "X-API-KEY: dxp_your_key"

# Get single pipeline
curl https://api.dataxpipe.com/api/v1/pipelines/orders_sync `
  -H "X-API-KEY: dxp_your_key"

The pipeline.json artifact includes name, schedule, owner, sources, targets, and check definitions. Registration is idempotent when the pipeline version matches; version bumps create a new catalog entry.

Runs

DAGs post run events at start, success, and failure.

# Create a run record
curl -X POST https://api.dataxpipe.com/api/v1/runs/ `
  -H "X-API-KEY: dxp_your_key" `
  -H "Content-Type: application/json" `
  -d '{"pipeline_id":"orders_sync","status":"running","started_at":"2025-06-01T06:00:00Z"}'

# Query runs for a pipeline
curl "https://api.dataxpipe.com/api/v1/pipelines/orders_sync/runs" `
  -H "X-API-KEY: dxp_your_key"

Run records link to check results and provide the audit trail operators use during incidents.

Checks

Post and query data quality check results.

# Execute a check against a registered connection
curl -X POST https://api.dataxpipe.com/api/v1/checks/execute `
  -H "X-API-KEY: dxp_your_key" `
  -H "Content-Type: application/json" `
  -d '{"connection_id":"bq-prod","sql":"SELECT COUNT(*) AS violations FROM ...","check_id":"chk_freshness","run_id":"run-abc123"}'

# Query recent check results
curl "https://api.dataxpipe.com/api/v1/checks/results?pipeline=orders_sync&status=fail" `
  -H "X-API-KEY: dxp_your_key"

Generated check scripts call these endpoints automatically. Set DATAXPIPE_URL and DATAXPIPE_API_KEY in your Airflow environment.

Connections

curl -X POST https://api.dataxpipe.com/api/v1/connections/ `
  -H "X-API-KEY: dxp_your_key" `
  -H "Content-Type: application/json" `
  -d '{"id":"pg-prod","type":"postgres","config":{"dsn":"postgresql://..."}}'

Supported connector types include postgres, bigquery, snowflake, and sqlite. Use _secret suffix keys in config to reference Vault or env-backed secrets.

Lineage

Query upstream and downstream dependencies for any dataset ID.

curl https://api.dataxpipe.com/api/v1/lineage/clean_orders `
  -H "X-API-KEY: dxp_your_key"

Lineage edges are derived from transform inputs and target declarations in the pipeline spec. Use this before schema changes to identify affected downstream marts and checks.

Organizations and billing

SaaS deployments expose organization management and Stripe billing endpoints:

GET /api/v1/organizations/me — current org, plan, and limits
POST /api/v1/billing/checkout — create Stripe Checkout session
POST /api/v1/billing/portal — open Customer Portal

Health and metrics

Endpoint	Purpose
`GET /health`	Liveness probe
`GET /ready`	Readiness (includes DB connectivity)
`GET /metrics`	Prometheus metrics when enabled

Set DATAXPIPE_URL, DATAXPIPE_API_KEY, and DATAXPIPE_TIMEOUT_SECONDS on all generated clients.

For a full walkthrough, see Getting Started with DataXPipe.