dagster-aws

dagster-aws is the official AWS integration package for the Dagster orchestration platform, maintained as a subpackage of the dagster-io/dagster monorepo. It bundles configurable resources, IO managers, a run launcher, and Pipes clients for the AWS services most commonly used in data pipelines: S3, ECS, EMR (classic, Serverless, and on EKS), Glue, Lambda, Redshift, Athena, RDS, Secrets Manager, Systems Manager Parameter Store, CloudWatch, and ECR.

Rather than hand-rolling boto3 calls inside every op, teams use dagster-aws’s ConfigurableResource classes to inject typed AWS clients into ops and assets, its S3-backed IO managers to persist intermediate outputs, its ECS run launcher to execute Dagster runs as isolated ECS tasks, and its Pipes clients to launch and monitor external compute (EMR jobs, Glue jobs, Lambda functions) from within a Dagster pipeline while streaming logs and metadata back through CloudWatch.

What You Get

• S3Resource, a pickled-object S3 IO manager, and an S3-backed compute log manager for persisting outputs, artifacts, and structured logs to S3
• An ECS run launcher and executor for running each Dagster run as its own isolated ECS task
• Pipes clients for ECS, EMR, EMR Serverless, EMR on EKS, Glue, and Lambda that launch external jobs and stream their logs and results back into Dagster via CloudWatch
• ConfigurableResource wrappers for Redshift, Athena, RDS, Secrets Manager, SSM Parameter Store, and ECR
• dg-cli component classes (S3ResourceComponent, RDSResourceComponent, SSMResourceComponent, and others) for declaring AWS resources through Dagster’s YAML/component-based project scaffolding
• Full boto3 session configuration (profile, region, endpoint, retries, unsigned requests) exposed as typed, validated Pydantic fields on every resource

Common Use Cases

• Persisting asset and op outputs to an S3 bucket via the S3 IO manager instead of local disk
• Launching a Spark job on EMR Serverless or EMR on EKS from a Dagster asset and waiting for completion via Pipes
• Running Dagster job executions as ephemeral ECS tasks for workload isolation and elastic scaling
• Reading secrets and configuration from AWS Secrets Manager or SSM Parameter Store into resource configuration at runtime
• Loading and unloading data from Redshift or Athena as part of an ELT pipeline

Under The Hood

Architecture dagster-aws is organized as a flat collection of AWS-service-scoped subpackages (s3, ecs, emr, athena, redshift, secretsmanager, ssm, cloudwatch, ecr, rds, pipes, components) under one top-level package, each independently importable (e.g. dagster_aws.s3, dagster_aws.ecs). Resources subclass Dagster’s ConfigurableResource (Pydantic-based) and IAttachDifferentObjectToOpContext to expose typed boto3 clients to ops and assets; IO managers subclass UPathIOManager/ConfigurableIOManager; the ECS run launcher subclasses Dagster’s RunLauncher/ConfigurableClass and constructs and monitors ECS tasks directly via boto3; the pipes subpackage layers a PipesClient abstraction (context injectors plus CloudWatch-backed message readers) over each AWS compute service so external jobs can report structured results back to the run. A components subpackage further wraps several resources as dg-cli Component classes registered via project entry points, letting users declare AWS resources through Dagster’s newer component-based scaffolding rather than Python code. The module boundary is AWS-service, not Dagster-primitive-type, so the layout grows additively as new services are added rather than through a shared abstraction layer.

Tech Stack Python (>=3.10,<3.15) built with hatchling; core runtime dependencies are boto3, the sibling dagster core package (pinned to the same monorepo version via a uv editable workspace source), packaging, and requests. Optional extras add psycopg2-binary for Redshift and dagster-pyspark for PySpark-on-EMR workflows, plus a stubs extra pulling in boto3-stubs-lite scoped to s3/logs/ecs/glue/emr* for static typing. Test dependencies (moto’s mocked AWS services, requests-mock, xmltodict, pytest-rerunfailures, pytest-cases) run through tox with a uv-locked venv runner. Linting is centralized at the monorepo root via ruff, with a package-local ruff.toml adding a banned-import rule that forbids top-level imports of mypy_boto3_* stub modules so they only load under TYPE_CHECKING.

Code Quality dagster_aws_tests mirrors the source tree one-for-one (s3_tests, ecs_tests, emr_tests, athena_tests, redshift_tests, rds_tests, secretsmanager_tests, ssm_tests, cloudwatch_tests, pipes_tests, components_tests), and tests run against moto’s mocked AWS services rather than live infrastructure, keeping them fast and hermetic. Source uses Dagster’s own _check module for explicit runtime parameter validation rather than silently trusting inputs, and resource classes lean on Pydantic for declarative, typed configuration with field-level descriptions. Boto3 stub imports are deliberately deferred to keep the runtime import graph light, and the monorepo’s root pyproject.toml centralizes lint/CI configuration across all python_modules packages.

API Design The package’s core value is turning a grab-bag of boto3 calls into first-class Dagster building blocks with consistent shapes across a dozen AWS services: every resource is a typed ConfigurableResource with Pydantic-validated fields (region, profile, retries, endpoint overrides) rather than a bag of kwargs, every launch-and-wait integration (ECS, EMR, EMR Serverless, EMR on EKS, Glue, Lambda) is exposed through the same PipesClient interface so switching compute backends barely changes calling code, and the CloudWatch-backed message reader gives all of them a uniform way to stream logs and metadata back into the Dagster UI. The components layer additionally lets these resources be declared via YAML-based scaffolding instead of hand-written Python, lowering boilerplate for common setups. None of this is technically novel relative to boto3 itself, but the DX payoff of one consistent mental model across a dozen AWS services is a genuine ergonomics win for teams standardizing on Dagster.