Kestra

Kestra is an open-source, event-driven orchestration platform that unifies scheduled and real-time workflow automation behind a single declarative YAML interface. Designed for data engineers, DevOps teams, and platform engineers, it lets you build, version, and run complex pipelines for data ingestion, infrastructure automation, AI agent workflows, and microservice orchestration without being tied to a specific programming language or runtime.

At its core, Kestra treats every workflow definition as code that lives in Git. Changes made through the built-in UI editor, the REST API, Terraform, or CI/CD pipelines are always reflected in the YAML source — so your orchestration logic is always auditable and reproducible. The platform ships with over 1,200 plugins covering cloud providers (AWS, GCP, Azure), message brokers (Kafka, Pulsar, Redis, NATS, SQS, Google Pub/Sub), databases, scripting runtimes (Python, Node.js, R, Go, Shell), and developer tools.

Kestra’s execution model is built for scale and resilience. It uses an internal queue backed by JDBC (PostgreSQL, MySQL, H2) or Kafka for message passing between components, with a dedicated executor, scheduler, worker, and indexer that can each be scaled independently. Task runners extend execution to Docker containers, Kubernetes jobs, and remote SSH hosts — letting compute-intensive steps run where they make sense without redesigning the workflow.

The platform now includes native MCP (Model Context Protocol) server support, allowing AI agents to invoke Kestra flows as tools. Combine this with the Langchain4j integration, built-in subflow composition, namespace-level isolation, and a live Directed Acyclic Graph topology view, and Kestra becomes a control plane that bridges traditional data engineering, infrastructure automation, and AI-native workloads.

What You Get

• Declarative YAML workflow definitions that stay synchronized whether you edit through the UI, API, or CI/CD pipeline
• A built-in code editor with real-time syntax validation, auto-completion, and a live DAG topology viewer
• Over 1,200 plugins covering AWS, GCP, Azure, Kafka, Pulsar, Redis, NATS, SQS, Google Pub/Sub, PostgreSQL, MySQL, and more
• Task runners that execute workloads in Docker containers, Kubernetes jobs, or remote SSH hosts without changing the workflow definition
• Native Git integration for version control, with a Terraform provider for managing flows as infrastructure
• Namespace-level isolation for organizing flows, secrets, and variables across teams and environments
• Event-driven triggers reacting to file arrivals, message broker events, webhooks, and inter-flow dependencies
• Native MCP server support exposing Kestra flows as tools for AI agents via the Model Context Protocol

Common Use Cases

• Data pipeline orchestration — data engineers build ETL/ELT workflows that extract from APIs or databases, transform with Python or dbt, and load into data warehouses like BigQuery or Snowflake
• Infrastructure automation — DevOps teams automate provisioning, configuration drift detection, and incident response workflows across AWS, GCP, and Azure environments
• AI agent workflows — platform engineers expose business workflows as MCP tools so that LLM-based agents can trigger Kestra flows for data retrieval, processing, and reporting
• Microservice coordination — backend teams orchestrate sequences of HTTP API calls, Kafka message publishing, and database updates with retry logic and error handling built in
• Scheduled batch processing — analytics teams run nightly or hourly batch jobs with cron-style scheduling, backfill support, and execution history tracking
• Event-driven automation — teams react to S3 file uploads, Kafka topic messages, or GitHub webhooks to trigger downstream processing pipelines automatically

Under The Hood

Architecture Kestra is built as a modular, event-driven system with well-separated components: a CLI entry point, a core domain model, a REST webserver, an executor engine, a scheduler, one or more workers, and an indexer — each runnable as independent services or collapsed into a single standalone process for simpler deployments. The execution engine uses an internal message queue as the backbone for passing workflow state between components, with the queue backed by either a JDBC store (PostgreSQL, MySQL, H2) or Kafka for high-throughput scenarios. The Flow model — represented as a directed acyclic graph of Tasks with Triggers at the boundary — is the central abstraction, and all mutations to it (from UI, API, Terraform, or CI/CD) converge on the same YAML serialization, ensuring the declaration is the single source of truth at all times.

Tech Stack Kestra is implemented in Java 25, built with Gradle in a multi-module workspace, and relies heavily on the Micronaut 4 framework for dependency injection, configuration management, and HTTP server capabilities. Persistence is handled through JDBC abstractions that support PostgreSQL, MySQL, and H2, with Flyway managing schema migrations. Worker-to-controller communication uses gRPC with Protocol Buffers for efficient binary serialization. The frontend is a Vue.js single-page application bundled with Vite. The build system applies Spotless for code formatting, SonarQube for static analysis, OWASP dependency checking, and JaCoCo for coverage aggregation. Cloud SDK BOMs from Google Cloud, Azure, and AWS are pinned platform-wide to keep plugin dependencies aligned.

Code Quality Kestra maintains an extensive test suite with close to 700 test classes distributed across all major modules, using JUnit 5 with custom Kestra-specific test annotations and the AssertJ assertion library. The CI pipeline covers backend tests (with Codecov and SonarQube integration), frontend tests, CodeQL static analysis, and a dedicated end-to-end scheduling test workflow. The codebase uses Lombok to reduce boilerplate, enforces code formatting via Spotless conventions applied through a shared Gradle convention plugin, and publishes OpenMetrics-compliant Prometheus metrics following a documented naming standard. Error handling is explicit throughout the execution model, with typed exceptions, retry abstractions at the task level, and separate error task lists per flow.

What Makes It Unique Kestra’s most distinctive quality is the bidirectional synchronization between the YAML definition and every editing surface — the web UI, the REST API, the Terraform provider, and CI/CD pipelines all write back to the same canonical YAML, making the declarative definition genuinely the single source of truth rather than a secondary export. The native MCP server integration is a significant differentiator: Kestra flows can be exposed directly as tools for LLM-based AI agents via the Model Context Protocol, bridging traditional workflow orchestration and AI-native workloads in the same platform. The plugin architecture — where each plugin module is independently packaged and loaded at runtime — allows the ecosystem to scale to over 1,200 integrations without bloating the core binary, and custom plugins follow the same interface as built-in ones.

Self-Hosting

Kestra is released under the Apache License 2.0, a permissive open-source license that allows commercial use, modification, distribution, and private use without requiring derivative works to be open-sourced. There are no copyleft obligations for self-hosters: you can run Kestra internally, integrate it into proprietary workflows, and modify the source without any obligation to publish your changes. The Apache 2.0 license also includes an explicit patent grant, which provides additional legal clarity for enterprise users.

Running Kestra yourself means taking ownership of the full operational stack. The minimum viable deployment is a single Docker container with an embedded H2 database, suitable only for local development. Production deployments require a PostgreSQL or MySQL instance, persistent object storage for task outputs and artifacts, and careful sizing of the executor, scheduler, worker, and webserver components — each of which can be scaled horizontally but must be monitored independently. High availability requires running multiple worker and executor instances behind a load balancer, configuring the internal queue for failure recovery, and setting up backup strategies for both the database and the storage layer. The engineering investment to operate Kestra reliably at scale is non-trivial and requires familiarity with JVM tuning, container orchestration, and the Kestra configuration model.

Kestra Technologies offers a cloud-hosted Enterprise Edition (Kestra Cloud) and an on-premises Enterprise Edition that add capabilities not present in the open-source core: role-based access control, audit logs, SSO and SAML integration, multi-tenancy, secrets management backends, enterprise-grade support SLAs, and managed upgrades. Self-hosters on the Apache 2.0 build give up these features along with Kestra’s managed infrastructure, automatic scaling, and direct vendor support. The open-source community can be reached via GitHub issues and the Slack workspace, but there is no guaranteed response time or production incident support outside a commercial contract.