dagster-postgres

dagster-postgres provides the PostgreSQL implementations of Dagster’s core instance storage interfaces: run storage, event log storage, and schedule storage. Instead of the default SQLite-backed storage meant for local development, teams point their dagster.yaml at PostgreSQL to get a durable, concurrent-safe backend for the Dagster webserver and daemon in production deployments.

Beyond basic connection-string configuration, the package includes a pluggable authentication layer for cloud-managed Postgres: Azure Workload Identity Federation, GCP Application Default Credentials, and AWS IAM database authentication via RDS auth tokens. These providers inject short-lived, auto-refreshing credentials directly into SQLAlchemy’s connection lifecycle, so operators can avoid static database passwords entirely on AKS, GKE, and EKS.

What You Get

• DagsterPostgresStorage - a single configurable class that wires up run, event log, and schedule storage against one PostgreSQL database from a dagster.yaml block
• Independent storage components - PostgresRunStorage, PostgresEventLogStorage, and PostgresScheduleStorage can be configured separately if you split storage across databases
• Alembic-managed schema migrations - versioned migrations under dagster_postgres/alembic keep the storage schema in sync across Dagster upgrades
• Cloud IAM/WIF token authentication - AzureWifTokenProvider, GcpWifTokenProvider, and AwsWifTokenProvider issue and auto-refresh short-lived database credentials instead of static passwords
• Connection resiliency helpers - retry_pg_connection_fn and retry_pg_creation_fn in utils.py handle transient connection failures with backoff
• StringSource/IntSource config interpolation - postgres_url and postgres_db fields can be sourced from environment variables in dagster.yaml

Common Use Cases

• Production Dagster deployments - swap the default SQLite instance storage for a durable, concurrent-safe PostgreSQL backend behind dagster-webserver and dagster-daemon
• Cloud-managed Postgres without static passwords - authenticate to Azure Database for PostgreSQL, Cloud SQL, or RDS using workload identity instead of long-lived credentials
• Kubernetes deployments on AKS/GKE/EKS - use the WIF token providers to pick up pod-projected identity tokens automatically via DefaultAzureCredential, ADC, or IRSA
• Split storage topologies - configure run storage, event log storage, and schedule storage against separate PostgreSQL instances for isolation or scaling
• Zero-downtime schema upgrades - rely on the bundled Alembic revisions to migrate storage schema in lockstep with Dagster core version upgrades

Under The Hood

Architecture DagsterPostgresStorage composes three SQL-backed storage classes (PostgresRunStorage, PostgresEventLogStorage, PostgresScheduleStorage), each extending shared base classes from dagster._core.storage (SqlRunStorage, EventLogStorage, ScheduleStorage) and implementing ConfigurableClass so they can be instantiated purely from a YAML block in dagster.yaml. A shared utils.py centralizes connection-string construction, engine/session creation, retry logic, and Alembic configuration reused by all three storage classes, while auth.py layers on a PgTokenProvider abstraction that hooks SQLAlchemy’s do_connect event to inject a fresh token as the connection password on every new DBAPI connection, keeping cloud-IAM auth transparent to the storage classes above it.

Tech Stack Python 3.10 through 3.14, built with hatchling and versioned in lockstep with dagster core via a 1!0+dev local version. Core runtime dependencies are dagster and psycopg2-binary; SQLAlchemy handles engine/connection pooling and query construction, and Alembic drives schema migrations from dagster_postgres/alembic. Optional extras (azure, gcp, aws, wif) pull in azure-identity, google-auth, and boto3 only when their respective token providers are used, keeping the base install lean. The package lives as a uv workspace member with editable path dependencies on sibling dagster, dagster-test, dagster-pipes, and dagster-shared packages within the monorepo.

Code Quality The test suite (dagster_postgres_tests) covers run storage, event log storage, schedule storage, daemon cursor storage, auth token providers, and full instance wiring, plus a dedicated compat_tests/test_back_compat.py suite that exercises migrations against older schema snapshots for backward compatibility. Validation throughout relies on dagster’s internal _check module for explicit runtime parameter checks alongside modern PEP 604 union type hints, and the package inherits the monorepo’s shared ruff/pyright configuration rather than defining its own linter setup.

API Design The primary entry point, DagsterPostgresStorage, is activated purely through declarative YAML configuration in dagster.yaml - no application code needs to reference the classes directly. StringSource and IntSource config fields allow every setting, including the connection URL, to be sourced from environment variables without extra plumbing. The WIF token providers eliminate the boilerplate of manual token refresh and password rotation that cloud IAM authentication would otherwise require from the operator.