Kestra is an open-source orchestration platform designed to automate and manage event-driven and scheduled workflows across data, infrastructure, and AI systems. It empowers engineers, data teams, and DevOps professionals to replace fragile glue code with declarative, version-controlled workflows that scale to enterprise workloads. By unifying scheduling, event triggers, and cross-platform integrations into a single engine, Kestra eliminates silos between teams and reduces operational complexity.
Built in Java and designed for high availability, Kestra supports deployment via Docker, Kubernetes, AWS CloudFormation, GCP Terraform, and more. Its architecture is API-first and Git-native, enabling CI/CD pipelines to manage workflows as code. With a rich plugin ecosystem and a visual UI that syncs with YAML definitions, Kestra ensures consistency between developer and non-developer workflows while maintaining full code control.
What You Get
- Declarative YAML Workflows - Define complex data pipelines, infrastructure tasks, and AI workflows using simple, versionable YAML files that auto-sync with UI changes.
- Event-Driven Triggers - Automate workflows in real-time using triggers from Kafka, Redis, AWS SQS, Google Pub/Sub, MQTT, webhooks, file arrivals, and custom events.
- 1,200+ Built-in Plugins - Integrate with databases, cloud services (AWS, GCP, Azure), APIs, data tools (dbt, Spark, Airbyte), messaging systems, and more without writing custom code.
- Visual Topology Editor - Build and visualize workflows as Directed Acyclic Graphs (DAGs) with drag-and-drop tasks, real-time syntax validation, and auto-completion.
- Git Version Control Integration - Automatically commit and push workflow changes from the UI to Git repositories, enabling code review, branching, and CI/CD pipelines.
- Task Runners for Any Language - Execute scripts in Python, Node.js, Go, Shell, R, or run Docker containers and Kubernetes jobs directly within workflows.
- High Availability & Scalability - Designed to handle millions of workflows with fault tolerance, retries, timeouts, SLAs, and distributed execution across clusters.
- RBAC, Audit Logs & SOC 2 Readiness - Enforce access controls, track changes, and meet enterprise compliance requirements for production-grade orchestration.
- AI-Powered Copilot & Agents - Use AI to generate, debug, and optimize workflows with natural language prompts, reducing manual coding effort.
- Terraform Provider - Manage Kestra resources (flows, triggers, namespaces) as infrastructure code using the official Terraform provider.
Common Use Cases
- Running dbt data pipelines - Data engineers use Kestra to schedule dbt models, run quality checks, and trigger Slack notifications—all in a single, version-controlled workflow.
- Automating cloud infrastructure audits - DevOps teams deploy nightly system checks across hybrid environments, upload logs to S3, and alert on failures using Kestra’s SSH and AWS plugins.
- Orchestrating AI/ML workflows - ML engineers automate RAG pipelines, model inference, evaluation, and retraining with Kestra’s Python and Kubernetes task runners and AI Copilot.
- Standardizing CI/CD and operational scripts - SRE teams replace scattered shell scripts with unified, auditable workflows for deployment, monitoring, and incident response across teams.
Under The Hood
Architecture
- Modular monolith design with clearly separated components (core, webserver, processor, storage, repository) implemented as independent Maven/Gradle modules
- Dependency injection via Micronaut enables pluggable implementations for storage, metadata, and persistence layers
- Annotation-driven cross-cutting concerns (retry logic, plugin registration) keep business logic clean and testable
- Configuration-driven backends allow seamless switching between storage and database providers without code changes
- Extensible templating and namespace injection systems support dynamic expression evaluation and sandboxed file operations
Tech Stack
- Java-based backend with modular architecture, built using Gradle and running on Eclipse Temurin JRE
- PostgreSQL as primary database with JDBC support for MySQL, storing metadata, queues, and storage state
- Docker-based deployment with rootless containers and Docker-in-Docker for isolated task execution and persistent volume mounting
- OpenAPI 3.0-defined REST APIs with embedded HTTP server for flows, executions, and AI-generated workflows
- Pebble templating engine integrated for dynamic YAML generation and expression evaluation
- CI/CD pipelines with GitHub Actions, Codecov integration, and Gradle-based executable JAR generation
Code Quality
- Comprehensive test coverage across unit, integration, and end-to-end scenarios with expressive assertions using JUnit 5 and AssertJ
- Declarative error handling via custom annotations that encapsulate retry logic, backoff strategies, and exception filtering
- Strong type safety and immutability enforced through well-defined interfaces and schema-backed YAML validation
- Consistent naming, clear layer separation, and modular boundaries enable maintainable extension and debugging
- Robust UI testing with Storybook and Vue 3, using real HTTP mocks to validate complex form interactions and state transitions
What Makes It Unique
- Code-first workflow definitions in YAML/JSON enable version-controlled, CI/CD-native pipeline management unlike GUI-centric tools
- Advanced retry annotations with predicate-based failure handling and environment-aware parameters reduce boilerplate while increasing fault tolerance
- Virtual node-based task dispatching with async event streaming allows horizontal scaling without centralized bottlenecks
- Real-time visual editor that generates interactive flow graphs directly from source code, preserving declarative syntax
- Compile-time validation via custom JSR-303 annotations for complex workflow constraints, ensuring type-safe pipeline definitions
- Unified data interface across execution, storage, and UI layers ensures consistent serialization and rich interactive experiences