Arroyo

Arroyo is an open-source distributed stream processing engine written in Rust that lets you write standard SQL to run stateful, real-time computations over both bounded and unbounded data — turning what would otherwise be a batch job into a pipeline that emits results the moment data arrives. It targets the same class of problem as Apache Flink or Kafka Streams, but ships as a single self-contained binary (or Docker image / Helm chart) rather than requiring a JVM cluster, and treats SQL as a first-class interface instead of an afterthought bolted onto a lower-level API.

Under the hood, submitted queries are parsed and optimized by a planner built on Apache DataFusion and Arrow, then compiled into a dataflow graph that Arroyo’s controller schedules across a pool of worker processes. Pipelines can maintain windows, joins, and other stateful operators across upstream failures because Arroyo checkpoints operator state directly to object storage (S3-compatible or similar) using a Parquet-based backend, giving pipelines fault tolerance and the ability to rescale without losing progress. For logic that doesn’t fit neatly into SQL, Arroyo supports user-defined functions in both Rust and Python, dynamically compiled and loaded by a dedicated compiler service at pipeline build time.

Arroyo ships with a wide range of built-in connectors — Kafka, Confluent, Kinesis, Fluvio, MQTT, NATS, Redis, RabbitMQ, webhooks, WebSockets, and filesystem/Iceberg sinks among them — so pipelines can read from and write to the systems most real-time architectures already use, without writing custom I/O code. A bundled web console (built as a separate TypeScript application) provides a visual pipeline graph, a SQL/Monaco-based query editor, and job monitoring, so operators aren’t limited to the CLI for day-to-day use.

The project is dual-licensed under Apache 2.0 and MIT with no enterprise-only fork or license-gated feature set, and it has attracted an active open-source community with regular releases and ongoing commit activity, backed by ArroyoSystems, the company behind it.

What You Get

• A SQL-first stream processing engine that runs stateful windows, joins, and aggregations over unbounded data with subsecond result latency.
• Built-in fault tolerance via Parquet-based checkpointing to object storage, so pipelines can recover and rescale without losing state.
• A broad connector library out of the box — Kafka, Confluent, Kinesis, Fluvio, MQTT, NATS, Redis, RabbitMQ, webhooks, WebSockets, and Iceberg/filesystem sinks.
• Support for user-defined functions in both Rust and Python, compiled and loaded dynamically by a dedicated compiler service.
• A bundled web console with a visual pipeline graph, a Monaco-based SQL editor, and job monitoring.
• Multiple deployment options — a single self-contained binary, an official Docker image, or a Helm chart for Kubernetes.

Common Use Cases

• Detecting fraud and security incidents in real time by scoring event streams as they arrive.
• Powering real-time product and business analytics dashboards instead of waiting on batch ETL windows.
• Streaming ingestion into a data warehouse or data lake with transformations applied in flight.
• Generating real-time features for machine learning models from live event streams.

Under The Hood

Architecture Arroyo’s CLI entry point dispatches into distinct service roles — API, Controller, Compiler, Worker, Node, and a combined Cluster mode — each backed by dedicated crates. User-submitted SQL is parsed and planned by the planner crate, built on top of Apache DataFusion and Arrow, producing a logical dataflow graph that the controller schedules across worker processes. State and checkpoints are persisted through a Parquet-based backend to pluggable object storage using a job/epoch/operator path layout, while gRPC contracts connect the controller and workers. A separate compiler service dynamically compiles user-defined functions into loadable code at pipeline-build time, and a pluggable connector layer covers a comprehensive range of sources and sinks including Kafka, Kinesis, Confluent, Fluvio, MQTT, NATS, Redis, RabbitMQ, filesystem/Iceberg destinations, webhooks, and other streaming protocols. Pipeline and job metadata lives in Postgres, managed through versioned SQL migrations.

Tech Stack The engine is a large multi-crate Rust workspace built on Apache Arrow and DataFusion for the SQL query engine and columnar batch processing, Tonic-based gRPC alongside Tower and Axum for internal service communication and HTTP APIs, and a Parquet-backed object storage layer for checkpoint state that can also write out to lakehouse formats via Delta Lake integration. The control plane persists metadata in Postgres through a connection-pooled client with generated query bindings, secures transport with a modern TLS stack, and uses a production-grade allocator for performance-sensitive workloads. An optional Python UDF path built on PyO3 lets non-Rust users extend pipelines without leaving SQL. The web console is a separate TypeScript application using Chakra UI, a Monaco-based query editor, and D3/dagre for pipeline graph visualization, with its API client generated directly from the Rust service’s OpenAPI schema. Deployment options span a single self-contained binary, an official Docker image, and a Helm chart for Kubernetes.

Code Quality The project maintains an extensive test suite spanning unit tests embedded across crates and dedicated testing crates for SQL smoke tests, UDF behavior, and end-to-end API integration. Continuous integration enforces formatting and linting on both the Rust workspace and the TypeScript console, plus a dedicated static-analysis security scan, indicating real quality gates rather than a token setup. Errors are modeled through a custom domain-specific error type with retry-hint metadata generated via a macro, and a well-known Rust error-handling library is used consistently across the control-plane and storage crates rather than relying on ad hoc panics. Naming and module boundaries follow idiomatic Rust conventions, with each crate scoped to a single bounded concern. Inline comments are sparse throughout the codebase, leaning on Rust’s type system and function-level documentation over prose — typical of an advanced systems codebase, but something that raises the learning curve for newcomers.

What Makes It Unique Unlike SQL-on-streams tools that wrap a JVM-based engine or require a fully hosted service, Arroyo compiles submitted SQL directly into native dataflow pipelines and dynamically compiles user-defined functions — including Python UDFs — into loadable code at build time through its own compiler service, enabling JVM-free, single-binary deployment with subsecond latency. Its checkpoint backend writes directly to object storage rather than requiring a dedicated state store, letting pipelines rescale and recover in a serverless-friendly way in modern cloud environments. By building its SQL planner on a well-established columnar query engine rather than reinventing an optimizer from scratch, Arroyo gets a modern execution path while remaining a purpose-built streaming runtime rather than a batch engine retrofitted for streaming.

Self-Hosting

Licensing Model

Arroyo is dual-licensed under Apache License 2.0 and MIT, the standard permissive licensing scheme used by most Rust projects. Both license files (LICENSE-APACHE, LICENSE-MIT) are included in full at the repository root, and there is no separate enterprise or “ee” license layered on top.

Self-Hosting Restrictions

None found. There are no ee/, enterprise/, or pro/ directories in the codebase, and no license-key checks, feature gates, or isEnterprise/requiresLicense-style code were found anywhere in the source. The full engine — SQL planner, connectors, checkpointing, and web console — ships as open source and can be self-hosted without restriction, via a downloaded binary, Docker image, or the included Helm chart.

Enterprise Features

There is no separate enterprise tier described in the repository. All capabilities present in the codebase (connectors, UDFs, checkpointing, the web console) are part of the single open-source distribution.

Cloud vs Self-Hosted

Arroyo does not operate its own first-party managed cloud offering documented in this repository. Instead, the README points to Cloudflare Pipelines, a separate managed service built on the Cloudflare Developer Platform that packages Arroyo-style stream processing as a hosted product; this is a third-party integration rather than an Arroyo-run SaaS, and using it is entirely optional — the self-hosted engine is fully functional on its own.

License Key Required

No. Nothing in the codebase, build process, or CLI requires a license key or account registration to run.