Getting Started
Learn how to define declarative pipeline specs, generate Airflow DAGs and SQL transforms, register metadata with the Catalog API, and run your first data quality checks.
Read article →DataXPipe Blog
Pipeline engineering, lineage & data quality
Guides for data teams building declarative pipelines, metadata catalogs, and production-grade checks — from the engineers behind dataxpipe.com.
All articles
15 guides for data engineering teams
- Data Quality
BigQuery Freshness Checks in DataXPipe
Implement freshness checks against BigQuery tables, configure thresholds in pipeline specs, and post structured results to the DataXPipe Catalog.
- #bigquery
- #freshness
- #data-quality
- Best Practices
Metadata Catalog Buyer's Guide 2025
Evaluate pipeline metadata catalogs for your data platform with criteria for lineage, check integration, API coverage, and declarative spec support.
- #metadata
- #catalog
- #evaluation
- Deployment
SaaS Billing Setup with Stripe
Configure Stripe Checkout, Customer Portal, and webhooks for DataXPipe SaaS plans including Free, Team, and Business tiers with org-scoped entitlements.
- #stripe
- #billing
- #saas
- Getting Started
YAML Spec Validation Deep Dive
Understand DataXPipe JSON Schema validation, semantic rules, CI integration, and common spec errors that block artifact generation and catalog registration.
- #yaml
- #validation
- #getting-started
- Data Quality
Monitoring Pipeline Runs in DataXPipe
Track run status, correlate check failures, set up alerting, and use Catalog APIs and Prometheus metrics to observe pipeline health in production.
- #monitoring
- #observability
- #runs
- Integrations
Snowflake Connector Guide for DataXPipe
Configure Snowflake connections in the DataXPipe Catalog, register credentials securely, and run transform checks against Snowflake warehouses.
- #snowflake
- #connectors
- #integrations
- Best Practices
Multi-Tenant Pipeline Catalogs
Design organization-scoped pipeline catalogs with API keys, plan limits, and isolation patterns for SaaS deployments serving multiple data teams.
- #multi-tenant
- #saas
- #organizations
- Best Practices
Declarative vs Imperative Pipeline Definitions
Compare declarative YAML specs and imperative orchestration code for data pipelines, and learn when DataXPipe's spec-first approach reduces drift and accelerates onboarding.
- #best-practices
- #pipeline-specs
- #architecture
- Deployment
Running the Catalog on Postgres in Production
Migrate the DataXPipe Catalog from SQLite to managed Postgres, apply Alembic migrations, tune connection pools, and configure backups for production workloads.
- #postgres
- #deployment
- #database
- Integrations
Airflow Integration Guide for DataXPipe
Deploy generated Airflow DAGs, wire Catalog run events, configure connections, and integrate check execution into your existing scheduler environment.
- #airflow
- #orchestration
- #integrations
- Deployment
Deploy DataXPipe on DigitalOcean
Step-by-step guide to deploying the DataXPipe Catalog API on DigitalOcean App Platform or DOKS with managed Postgres, Redis, and container registry.
- #deployment
- #digitalocean
- #kubernetes
- Catalog API
Catalog API Quick Reference
A concise reference for DataXPipe Catalog API endpoints covering pipelines, runs, checks, connections, lineage, and organization-scoped authentication.
- #catalog-api
- #rest-api
- #reference
- Data Quality
A Practical Guide to Data Quality Checks
Design SQL-based checks with appropriate severity levels, integrate results with the DataXPipe Catalog, and build alerting workflows that catch issues before stakeholders do.
- #data-quality
- #checks
- #observability
- Lineage
Pipeline Lineage Best Practices
Design dataset identifiers, model upstream/downstream dependencies, and expose lineage through the DataXPipe Catalog so impact analysis and debugging take minutes, not days.
- #lineage
- #metadata
- #catalog-api
Ready to catalog your pipelines?
Register an organization, get your API key, and start tracking pipelines, runs, and quality checks today.