dagster-cloud

dagster-cloud is the Python package that powers the Hybrid Agent for Dagster+ (formerly Dagster Cloud) — the process you run inside your own VPC, Kubernetes cluster, ECS cluster, or Docker host so that Dagster’s hosted control plane can launch and monitor your data pipelines without ever seeing your code or credentials. It polls the Dagster Cloud API for pending work, launches user code servers through a pluggable set of launchers, and reports heartbeats, logs, and run results back to the host.

Beyond the agent runtime, the package bundles the DagsterCloudInstance implementation, workspace and code-location management, a serverless/PEX packaging pipeline for shipping code without building images, agent-token auth for talking to the Dagster Cloud GraphQL API, and a dagster_insights module for pushing Snowflake/BigQuery cost and usage metrics into Dagster’s Insights product.

What You Get

• Hybrid Agent runtime - a long-running poller (agent/dagster_cloud_agent.py) that checks in with Dagster Cloud, launches user code servers, and forwards run requests to the right compute backend.
• Pluggable user-code launchers - built-in launchers for Kubernetes, ECS, Docker, and local process/PEX execution so the agent can run your code wherever it already lives.
• Serverless & PEX packaging - tooling under pex/ to build and upload PEX bundles so Dagster Cloud’s serverless compute can run your code without a custom Docker image.
• DagsterCloudInstance - a DagsterInstance implementation (instance/__init__.py) that proxies storage, run launching, and telemetry through the Dagster Cloud API instead of a local database.
• Insights integration - a dagster_insights module that attaches Snowflake/BigQuery query cost and usage metrics to asset materializations for the Dagster+ Insights UI.
• Anomaly detection hooks - GraphQL mutations and types under anomaly_detection/ for wiring asset checks into Dagster+‘s anomaly detection feature.

Common Use Cases

• Hybrid deployment - a data platform team runs the agent inside its own Kubernetes cluster so code and secrets never leave the company’s VPC, while Dagster+ hosts the UI and scheduler.
• Multi-cloud code locations - an org with pipelines split across ECS and on-prem Docker hosts runs one agent per environment, each launching only its own code locations.
• Serverless onboarding - a smaller team skips agent infrastructure entirely by packaging code as PEX bundles that Dagster+‘s serverless compute executes directly.
• Cost attribution - a data engineering team enables the Insights integration to tie Snowflake credit spend back to the specific assets and jobs that generated it.

Under The Hood

Architecture The agent is a layered polling system: DagsterCloudAgent (agent/dagster_cloud_agent.py) runs a CHECK_WORKSPACE_INTERVAL_SECONDS loop that queries the Cloud GraphQL API (agent/queries.py) for workspace entries and pending run/step requests, dispatches them onto a FuturesAwareThreadPoolExecutor, and hands execution off to a DagsterCloudUserCodeLauncher implementation chosen per code location (Kubernetes, ECS, Docker, or local process). Storage and run-launching calls are routed through DagsterCloudInstance (instance/init.py), which subclasses Dagster’s own DagsterInstance so the rest of the Dagster core codebase is unaware it’s talking to a remote control plane rather than a local database — a clean substitution point that means changing the transport layer wouldn’t require touching launcher or workspace code.

Tech Stack Pure Python (3.9–3.15), built with hatchling, and pinned in lockstep to exact versions of dagster, dagster-shared, and dagster-cloud-cli. It layers typer and questionary for CLI ergonomics, pex for portable code packaging, requests/urllib3 for HTTP, and optional extras (docker, kubernetes, dagster-k8s, dagster-aws/boto3 for ECS) that pull in only the launcher backend actually in use, keeping the base install lean.

Code Quality The public mirror ships no test files at all — conftest.py references a dagster_cloud_test_infra pytest plugin that lives in Dagster’s internal monorepo, not here, so test coverage can’t be assessed from this repo alone. The code itself is consistently type-annotated (a py.typed marker is present), uses Dagster’s own _check runtime-assertion helpers pervasively, and wraps failures in structured SerializableErrorInfo objects rather than swallowing exceptions; a tests extra listing mypy, pylint, and pytest implies these checks run in Dagster’s private CI rather than being visible publicly.

API Design Installing and running the agent means installing one pip package, writing a small dagster_cloud.yaml/instance config, and running a single agent process per environment — deliberately little boilerplate for what is a distributed system. The tradeoff is that the package surface is broad (agent, instance, workspace launchers, insights, anomaly detection all in one distribution), so newcomers need to know which submodule matters for their deployment mode rather than importing one obvious entry point.