YAML Pipeline Spec for Postgres and Neon: Smoke Test in 10 Minutes

Postgres and Neon are the fastest path to a real warehouse connection when evaluating DataXPipe. One YAML spec declares credentials, sources, targets, and checks — then registers everything into your org catalog.

This walkthrough matches the Load example template in the app (neon_smoke_test).

Why connections live in the spec

DataXPipe treats warehouse credentials as part of the pipeline contract:

connections:
  - id: neon_main
    type: postgres
    config:
      host: YOUR-NEON-HOST.neon.tech
      database: neondb
      sslmode: require

sources:
  - id: raw_orders
    type: postgres
    connection_ref: neon_main
    object: public.orders
    load_mode: full

targets:
  - id: mart_orders_daily
    type: postgres
    connection_ref: neon_main
    object: public.mart_orders_daily
    write_mode: append

After registration, connections appear on the Connections page for the org. Sources and targets reference connection_ref instead of duplicating host strings.

Minimal smoke-test spec

pipeline:
  name: neon_smoke_test
  description: Smoke test pipeline
  owner: data-platform
  environment: dev
  schedule: "0 6 * * *"

connections:
  - id: neon_main
    type: postgres
    config:
      host: YOUR-NEON-HOST.neon.tech
      database: neondb
      sslmode: require

sources:
  - id: raw_orders
    type: postgres
    connection_ref: neon_main
    object: public.orders

targets:
  - id: mart_orders_daily
    type: postgres
    connection_ref: neon_main
    object: public.mart_orders_daily

checks:
  - id: freshness_mart
    type: freshness
    target: mart_orders_daily
    max_delay_minutes: 1440

Replace the host with your Neon connection string host. Use sslmode: require for Neon pooler/direct endpoints.

Register in the app

1. Sign in as platform admin
2. Upload YAML or Create pipeline
3. Click Load example or paste the spec above
4. Validate — fixes schema/semantic errors before register
5. Register pipeline — watch usage: first register uses 1 of 2 free pipelines

Re-registering the same pipeline.name updates the entry without using another pipeline slot.

Validate before register

Validation checks JSON Schema structure and semantics (required fields, valid references). Developers without register permission can still validate — useful for review before a platform admin registers.

CI can call the same validate endpoint:

curl -X POST https://api.dataxpipe.com/api/v1/specs/validate \
  -H "Content-Type: application/json" \
  -d '{"spec_yaml": "... paste yaml ..."}'

From smoke test to production

Stage	Action
Smoke test	Example spec, dev environment tag
Second pipeline	Add real ingest pipeline with second pipeline.name
Secrets	Prefer env-backed secrets in production deploy; document in spec owner field
Team	Invite developers; they validate, admin registers

Free tier limits (honest expectations)

Free includes 2 pipelines and 2 connections. The UI shows quota and blocks registering a third new pipeline name while still allowing updates to existing names.

Upgrade on the Billing page when you outgrow limits.

Related reading

• Getting Started with DataXPipe
• YAML spec validation deep dive
• Postgres catalog in production
• Use cases: YAML registration

Open Create pipeline and load the example to try this in your workspace.