Deploy DataXPipe on DigitalOcean

DigitalOcean provides a practical launch path for the DataXPipe SaaS backend: managed Postgres for the Catalog, Redis for Celery task queues, and either App Platform or Kubernetes (DOKS) for the FastAPI service. This guide covers both options with production-hardening defaults.

Architecture overview

Component	DigitalOcean service	Purpose
Catalog API	App Platform or DOKS	FastAPI, billing webhooks, OIDC
Metadata store	Managed Postgres 15	Pipelines, runs, orgs, lineage
Task queue	Managed Redis	Celery background jobs
Container images	DOCR	registry.digitalocean.com/<namespace>/dataxpipe

Frontends (marketing, product UI, blog) deploy separately on Vercel and call api.dataxpipe.com.

Option A: App Platform (recommended for launch)

App Platform handles TLS, scaling, and deploy hooks without managing cluster infrastructure.

1. Provision managed services

Create Managed Postgres and Managed Redis in the same region as your app. Note connection strings with sslmode=require for Postgres.

2. Build and push the container

docker build -t registry.digitalocean.com/myorg/dataxpipe:latest .
doctl registry login
docker push registry.digitalocean.com/myorg/dataxpipe:latest

3. Create the App

Connect your GitHub repo, set Dockerfile path to Dockerfile, and configure environment variables:

DATAXPIPE_DB=postgresql://user:pass@host:25060/dataxpipe?sslmode=require
DATAXPIPE_USE_ALEMBIC=true
DATAXPIPE_REDIS_URL=rediss://default:pass@host:25061
DATAXPIPE_CORS_ORIGINS=https://dataxpipe.com,https://app.dataxpipe.com
STRIPE_SECRET_KEY=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...
OIDC_ISSUER=https://your-idp.example.com
OIDC_AUDIENCE=dataxpipe
DATAXPIPE_RATE_LIMIT_ENABLED=true

4. Run migrations

Add a pre-deploy job or run once after first deploy:

python scripts/run_migrations.py

Set DATAXPIPE_USE_ALEMBIC=true so the app skips implicit schema initialization at startup.

5. Configure DNS

Point api.dataxpipe.com to the App Platform URL. Enable Let’s Encrypt in the App settings.

Option B: DigitalOcean Kubernetes (DOKS)

For teams that need custom networking, sidecars, or multi-region failover, deploy to DOKS using manifests in k8s/ and deploy/do/.

Prerequisites

• DO_API_TOKEN with read/write scope
• DOCR_REGISTRY namespace created
• kubectl configured for your cluster

Deploy workflow

The repository includes .github/workflows/deploy_digitalocean.yml which:

1. Builds and pushes the image tagged with the commit SHA
2. Updates k8s/deployment.yaml with the new image reference
3. Applies manifests to the cluster

Required GitHub secrets: DO_API_TOKEN, DOCR_REGISTRY, DO_K8S_CLUSTER_NAME.

Kubernetes secrets

Store sensitive values in Kubernetes Secrets rather than plain env vars:

# deploy/do/deployment-saas.yaml excerpt
env:
  - name: DATAXPIPE_DB
    valueFrom:
      secretKeyRef:
        name: dataxpipe-secrets
        key: postgres-url

Reference managed Postgres and Redis connection strings from DigitalOcean database dashboards.

Production hardening checklist

• TLS termination at the load balancer or ingress; bind uvicorn to localhost/internal interface only
• Secrets manager for warehouse credentials (DATAXPIPE_SECRETS_BACKEND=vault or env-backed _secret keys)
• Rate limiting via DATAXPIPE_RATE_LIMIT_ENABLED=true for single-instance; use Redis-based limiter for HA
• Backups with scripts/backup_db.py uploading to DigitalOcean Spaces
• Monitoring via /metrics endpoint scraped by Prometheus or DO monitoring

Stripe webhook configuration

After the API is live, register the webhook in Stripe Dashboard:

https://api.dataxpipe.com/webhooks/stripe

Subscribe to checkout.session.completed and customer.subscription.* events. Copy the signing secret to STRIPE_WEBHOOK_SECRET.

Verifying the deployment

curl https://api.dataxpipe.com/health
curl https://api.dataxpipe.com/ready
curl -X POST https://api.dataxpipe.com/api/v1/pipelines/ `
  -H "X-API-KEY: dxp_your_key" `
  -H "Content-Type: application/json" `
  -d @generated/orders_sync/metadata/pipeline.json

A healthy deployment returns 200 from /health, confirms Postgres connectivity on /ready, and accepts pipeline registration.

Next steps

Configure Vercel frontends per docs/deploy_saas.md and register warehouse connections for check execution.