Apache Airflow

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. Built at Airbnb and donated to the Apache Software Foundation, it treats pipelines as first-class Python code, giving data engineers the full power of the language to express complex task dependencies, parameterize runs, and version-control their pipelines alongside application code.

At the core of Airflow is the Directed Acyclic Graph (DAG) — a declarative representation of tasks and their dependencies. The scheduler continuously evaluates DAGs, queues tasks that are ready to run, and dispatches them to workers via pluggable executors: a local multiprocess executor for single-node deployments, a Celery executor for distributed queues, or a Kubernetes executor that launches each task as its own Pod. As of Airflow 3.0, the architecture has been refactored into a modular multi-package layout with a dedicated Task SDK, a clean FastAPI-based execution API, and first-class support for asset-driven scheduling.

Airflow ships with an extensive provider ecosystem covering over 80 external systems — AWS, GCP, Azure, Snowflake, dbt, Databricks, Spark, and many more — each providing pre-built operators, hooks, and sensors. The web UI offers real-time DAG graph visualization, task-level log streaming, backfill controls, and variable and connection management. Its plugin architecture allows teams to extend virtually every layer of the system without forking core code.

The project is maintained by a large and active Apache community with over 3,000 contributors and is in production at hundreds of organizations including Airbnb, LinkedIn, Twitter, ING, and NASA. It is the de facto standard for Python-native workflow orchestration in data engineering and MLOps.

What You Get

• DAG-based workflow authoring in pure Python with full support for dynamic task generation, branching, and parameterization via Jinja2 templating
• Pluggable executor backends — LocalExecutor for single-node, CeleryExecutor with Redis/RabbitMQ for distributed queues, KubernetesExecutor for per-task Pod isolation, and edge executors for multi-region deployments
• 80+ provider packages covering AWS, GCP, Azure, Snowflake, dbt, Databricks, Spark, Kafka, PostgreSQL, MySQL, HTTP, SSH, SFTP, Slack, and many more
• Built-in web UI with DAG graph visualization, Gantt charts, task log streaming, backfill management, and real-time run monitoring
• Asset-driven scheduling (Airflow 3.x) that triggers DAGs when upstream datasets or events are produced, enabling event-based pipeline orchestration
• Task SDK for decoupled task execution with a clean FastAPI-based execution API, enabling polyglot task definitions and sandbox-isolated runs
• XCom system for lightweight inter-task metadata exchange, with pluggable backends for large payloads
• Comprehensive security model with RBAC, per-DAG access control, secrets backends (Vault, AWS SSM, GCP Secret Manager), and audit logging

Common Use Cases

• ETL/ELT pipeline orchestration — data engineers schedule extract-transform-load jobs across databases, data lakes, and warehouses with full retry and alerting logic
• MLOps workflow automation — ML teams chain data prep, model training, evaluation, and deployment steps as DAGs with conditional branching on metric thresholds
• Daily business reporting — analytics teams run nightly aggregations, generate reports, and push results to BI tools or email on a cron-based schedule
• Data ingestion and API polling — sensors wait for external API endpoints or file arrivals in S3/GCS before triggering downstream processing tasks
• Infrastructure automation — platform teams use Airflow to orchestrate infrastructure provisioning, database migrations, and multi-step deployment workflows
• Cross-system data synchronization — operations teams keep data consistent across CRMs, ERPs, and data warehouses by scheduling incremental sync DAGs

Under The Hood

Architecture Airflow 3.x has completed a significant architectural evolution from its original monolithic design toward a cleanly layered, component-based system. The platform separates concerns across a Scheduler (responsible for heartbeating DAG runs, queuing task instances, and managing the state machine), a DAG Processor (which parses and serializes DAG files in isolation), an Execution API (a FastAPI service that mediates between the scheduler and executors), a Triggerer (for deferred async sensors), and pluggable Executor backends that actually run tasks. The Task SDK is a separately installable package that decouples the task execution contract from the core scheduler, enabling operators to be written and tested without a full Airflow install. Dependency injection is handled via SQLAlchemy sessions passed through a provide_session decorator pattern, and the DAG definition layer uses attrs-based dataclasses throughout, making the object graph composable and serializable to the metadata database.

Tech Stack Airflow is implemented in Python (3.10–3.14) with SQLAlchemy as the ORM against PostgreSQL or MySQL metadata backends (SQLite supported only for development). The web server uses FastAPI on Uvicorn/Gunicorn for the REST API and execution API layers, while the frontend is a React + Vite + TypeScript single-page application with Playwright for end-to-end tests. The Celery executor integrates with Redis or RabbitMQ as the task queue broker. Kubernetes support is provided through the official Python Kubernetes client. Structured logging is done via structlog, and observability metrics are emitted as StatsD-compatible gauges. Build tooling uses Hatchling, uv for dependency management, and pnpm for the frontend. CI runs on GitHub Actions across AMD64 and ARM64 platforms.

Code Quality Airflow’s test suite is comprehensive, with over 370 test files in the airflow-core package alone, organized into unit, integration, and system test tiers. The project enforces type annotations across the codebase, uses mypy for type checking, and has structured linting via ruff and eslint on the frontend. Error handling is explicit throughout — database operations use SQLAlchemy session management with retry decorators for transient failures, executor failures propagate through a typed state machine rather than being swallowed, and DAG parsing errors are stored as ImportError records in the metadata database rather than crashing the scheduler. The CI pipeline runs tests on two CPU architectures with separate AMD64 and ARM64 workflows, enforcing coverage and type checks on every PR.

What Makes It Unique Airflow’s defining innovation is that workflows are code, not configuration — a decision that gives pipelines the full testability, reusability, and version-control benefits of software. Unlike YAML-driven alternatives, dynamic DAG generation lets a single Python file produce hundreds of parameterized pipelines from a database query or external config. The asset-aware scheduling system introduced in Airflow 3.x is a significant advancement: DAGs can be triggered by the production of named data assets rather than wall-clock schedules, enabling true data-driven orchestration. The provider ecosystem’s breadth — 80+ integrations maintained by the community with a standardized hook/operator/sensor interface — means teams can compose multi-system pipelines without writing custom connectors. The deferrable operator model, where sensors yield control back to the Triggerer process while awaiting async events, eliminates the idle worker cost that plagued earlier sensor implementations.

Self-Hosting

Apache Airflow is released under the Apache License 2.0, one of the most permissive open-source licenses available. It allows unrestricted commercial use, modification, distribution, and sublicensing without copyleft obligations — you can embed it in proprietary products, modify the source code, and run it in a closed commercial environment with no requirement to share changes. The only obligations are attribution (keeping the Apache copyright notice) and not using Apache trademarks to imply endorsement. There are no open-core restrictions, no enterprise-edition source gating, and no license keys required for any feature.

Running Airflow yourself carries meaningful operational weight. A production-grade self-hosted deployment requires a PostgreSQL or MySQL metadata database (managed or self-operated), a message broker such as Redis or RabbitMQ if using the CeleryExecutor, and either persistent worker VMs or a Kubernetes cluster for the KubernetesExecutor. You are responsible for database backups, log storage (typically offloaded to S3 or GCS), scheduler high-availability (multiple schedulers are supported but require careful configuration), and rolling upgrades during version bumps — which can involve database schema migrations. The official Helm chart simplifies Kubernetes deployments significantly, but tuning worker concurrency, DAG file sync performance, and metadata database connection pooling still demands experienced operations staff.

The primary managed alternatives are Astronomer (the commercial company founded by core Airflow contributors, offering Astro Cloud and on-premise Astro Software), Amazon MWAA (Managed Workflows for Apache Airflow on AWS), Google Cloud Composer, and Azure Data Factory Airflow integration. These platforms trade configuration flexibility for managed upgrades, built-in HA, cloud-native log integration, and support SLAs. Astro in particular adds a UI for deployment management, DAG-level CI/CD, and observability features not present in open-source Airflow. Teams that need rapid onboarding or lack dedicated platform engineering capacity typically find the managed tax worthwhile; teams with existing Kubernetes expertise or strict data residency requirements often prefer self-hosting.