dagster-docker

dagster-docker is Dagster’s integration package for running orchestrated jobs inside Docker containers. It ships a DockerRunLauncher that starts each Dagster run in its own container, and a docker_executor that can run individual steps of a job in separate containers, giving pipelines container-level isolation without requiring a full Kubernetes deployment.

The package lives inside the Dagster monorepo alongside the other official integration libraries (dagster-k8s, dagster-celery-docker, dagster-aws, etc.) and depends directly on the core dagster package plus the docker Python SDK. It is typically installed alongside dagster and dagster-webserver in deployments that want to launch and execute Dagster runs as standalone Docker containers rather than in-process or via Kubernetes.

What You Get

• A DockerRunLauncher that starts each Dagster run inside a dedicated Docker container
• A docker_executor for running individual job steps in separate containers for isolation and parallelism
• Dagster Pipes integration (dagster_docker.pipes) for launching external Docker-based compute from a Dagster asset or op
• Container context configuration for env vars, networks, volumes, and registry/image overrides per run

Common Use Cases

• Isolating each Dagster run in its own container for reproducibility and dependency isolation
• Running individual pipeline steps as separate containers for resource isolation or parallel execution
• Launching external Dockerized compute jobs via Dagster Pipes without embedding that code in the Dagster process
• Deploying self-hosted Dagster where Kubernetes isn’t available but Docker is

Under The Hood

Architecture The package is organized around docker_run_launcher.py (the DockerRunLauncher that Dagster’s daemon calls to start a run as a new container via the Docker Engine API), docker_executor.py (the step-level executor that launches one container per step and streams logs back to the Dagster event log), container_context.py (resolves per-run Docker configuration such as networks, env vars, and registry credentials), and pipes.py (a thin wrapper that launches an external Docker container and bridges its stdout back into Dagster’s Pipes protocol). utils.py holds shared container lifecycle helpers (create, start, poll, tear down).

Tech Stack Pure Python, built with Hatchling, pinned to the same version as the core dagster package (1!0+dev in-repo). Its only non-Dagster runtime dependencies are the docker SDK (talks to the Docker Engine API) and docker-image-py (image reference parsing). It requires Python 3.10-3.14 and is developed with uv workspace sources pointing at sibling packages in the monorepo (dagster, dagster-pipes, dagster-k8s, etc.).

Code Quality Tests live in dagster_docker_tests/ and exercise the run launcher and executor against a real Docker daemon via conftest.py fixtures, following the same pytest-based testing conventions as the rest of the Dagster monorepo. Being part of a large, actively maintained monorepo with a shared CI pipeline, it benefits from Dagster’s broader linting, typing, and review standards rather than maintaining independent tooling.

API Design Configuration is done declaratively through Dagster’s run_config/@job(executor_def=docker_executor) patterns, consistent with how other Dagster executors (k8s, celery) are wired up, so switching between deployment targets is largely a config change rather than a code change. This keeps the integration’s surface area small and predictable for anyone already familiar with Dagster’s executor/run-launcher abstractions.