Quickwit

Quickwit is an open-source, cloud-native search engine built from the ground up for observability workloads—logs, distributed traces, and forthcoming metrics. Unlike traditional search engines that couple compute to expensive local SSD storage, Quickwit’s stateless architecture stores indexed splits on object storage like Amazon S3, Azure Blob, or Google Cloud Storage while delivering sub-second query latency through highly optimized IO paths built on the Tantivy full-text search library in Rust.

The project provides a drop-in Elasticsearch-compatible REST API, making it straightforward to point existing log shippers like Vector, Fluent Bit, or Logstash at Quickwit without reconfiguring pipelines. Native OpenTelemetry (OTLP) and Jaeger support means it integrates directly into modern observability stacks, ingesting traces and logs without middleware. A companion Grafana data source plugin rounds out visualization.

Quickwit scales horizontally with stateless indexers and searchers that can be deployed on Kubernetes via official Helm charts, run as a single binary for smaller deployments, or even executed as serverless functions on AWS Lambda for pay-per-query workloads. Its multi-tenant architecture handles multiple indexes with independent retention policies, GDPR delete tasks, and partitioned ingestion from Kafka, Kinesis, and Pulsar.

The engine is maintained by Datadog, ensuring high production hardening standards and long-term commitment. With over 11,000 GitHub stars, a broad contributor community, and an extensive documentation site spanning guides, tutorials, and API references, Quickwit has become the leading open-source alternative to proprietary observability storage backends.

What You Get

• Sub-second search on cloud storage - Queries execute directly against S3, Azure Blob, or GCS with custom-optimized IO paths and smart split-level pruning, eliminating the need for expensive local NVMe clusters or indexing nodes.
• Elasticsearch-compatible REST API - Supports the ES query DSL, field aggregations, and ingest endpoints, enabling seamless migration of existing Kibana dashboards, log shippers, and OpenSearch clients with zero code changes.
• Native OpenTelemetry and Jaeger integration - Ingests logs and traces via OTLP gRPC protocol and serves as a Jaeger-native storage backend, supporting 100% trace retention without sampling at cloud storage costs.
• Schemaless and strict schema indexing - Dynamically parses unstructured JSON logs out of the box while supporting optional schema enforcement for structured data with predictable storage layout and query performance.
• Decoupled compute and storage with stateless nodes - Indexers and searchers share no local state, allowing independent scaling of ingestion and query workloads and supporting zero-downtime restarts without data loss.
• Grafana data source plugin - An official Grafana plugin connects Quickwit indexes directly to Grafana dashboards for visualizing logs, traces, and aggregated time-series without additional data transformation.
• Multi-tenancy with isolated indexes - Supports multiple independent indexes with separate retention policies, delete tasks for GDPR compliance, and partitioned ingestion for isolated tenant workloads.
• Serverless AWS Lambda deployment - Purpose-built Lambda server and client crates enable pay-per-query serverless search at petabyte scale, scaling from zero cost at rest to full throughput on demand.

Common Use Cases

• High-volume log pipeline on object storage - A DevOps team replaces Elasticsearch with Quickwit to index 5TB/day of application logs from Vector and Fluent Bit, cutting storage costs by 80% while maintaining sub-second query latency across billions of events.
• Full distributed trace retention with Jaeger - A cloud-native platform uses Quickwit as the Jaeger storage backend to retain 100% of distributed traces for 90 days at S3 storage costs, enabling root-cause analysis across complete trace histories without sampling.
• Serverless search-as-a-service - A SaaS company deploys Quickwit on AWS Lambda to offer per-customer log search with near-zero infrastructure cost at rest, only incurring compute costs when customers actively query their data.
• Elasticsearch migration in a regulated industry - A financial services firm migrates from Elasticsearch to Quickwit to reduce licensing and infrastructure costs while implementing GDPR-compliant targeted deletion, keeping existing Kibana dashboards functional via the ES-compatible API.
• Multi-tenant observability platform - A managed services provider uses Quickwit’s multi-index architecture to host isolated log and trace storage for dozens of customers on shared infrastructure, with per-tenant retention policies and access boundaries.

Under The Hood

Architecture Quickwit is organized as a Rust Cargo workspace of roughly 30 fine-grained crates, each with a single well-defined responsibility—ingestion, indexing, search, metastore, storage, control plane, and cluster coordination. The actor framework in quickwit-actors provides the concurrency backbone: all stateful components are modeled as typed actors with mailboxes, supervision hierarchies, and observable health signals, making the pipeline composable and individually testable. A root search dispatcher fans queries out to stateless leaf searchers based on split placement, collects partial results, and merges aggregations—cleanly separating the coordination layer from the data execution layer. The split-based storage model (inherited from Tantivy) means all indexed data is immutable once uploaded, which enables safe concurrent access from multiple searcher nodes without locking and makes crash recovery trivially restartable.

Tech Stack The core is written in Rust using Tokio for async runtime, with the Tantivy full-text search library providing the inverted index, BM25 scoring, and aggregation primitives. gRPC via Tonic and Prost serves inter-node communication, while Axum and Warp power the REST API layer. Storage backends are unified behind a trait-based abstraction supporting Amazon S3, Azure Blob Storage, Google Cloud Storage via OpenDAL, and local filesystem. Metastore backends include PostgreSQL and a file-based implementation for single-node deployments. The embedded UI is built with React 19, Material UI, Monaco editor (for query autocomplete), and Vite, bundled and statically embedded into the main binary. CI runs on GitHub Actions with extensive matrix testing and cargo-nextest for fast parallel test execution.

Code Quality With over 455 files containing test attributes, the codebase has comprehensive unit, integration, and end-to-end test coverage. The quickwit-integration-tests crate spins up full cluster scenarios including OTLP ingestion, Kafka sources, and update workflows. Error handling throughout uses thiserror for typed domain errors and anyhow for propagation contexts, with explicit error variants at API boundaries to avoid silent failures. Clippy is enforced via deny(clippy::disallowed_methods) attributes, Rust formatting via rustfmt with a project config, and TypeScript tooling via Biome for the UI. The contributing guide requires tests for all PRs and a linked issue per commit, reflecting mature code-review culture.

What Makes It Unique Quickwit’s core innovation is rethinking the search index as an immutable, object-storage-native artifact. Rather than maintaining mutable local segments that require expensive SSDs and careful fsync management, each indexed split is sealed, checksummed, and uploaded to object storage immediately after creation. A custom byte-range IO scheduler minimizes cloud storage API calls during query execution by coalescing reads and caching hot footer metadata in memory. The result is that Quickwit can deliver sub-second search latencies over petabytes of cold data stored in S3-class storage—a combination previously achievable only with in-memory or local-SSD approaches—while running searcher nodes that can be killed and replaced in seconds without any data recovery procedures.

Self-Hosting

Quickwit is released under the Apache License 2.0, one of the most permissive open-source licenses available. It allows unrestricted commercial use, modification, distribution, and sublicensing without any copyleft obligations on application code—you can embed Quickwit in a commercial product or SaaS offering without being required to open-source your own code. The only obligations are preserving copyright notices and the NOTICE file. There are no enterprise-only feature tiers or proprietary add-ons gated behind a commercial license in the current codebase, meaning everything in the GitHub repository is fully available to self-hosters.

Running Quickwit yourself requires a working object storage bucket (S3, Azure Blob, or GCS), a metastore backend (PostgreSQL for production or file-based for single-node testing), and compute resources for indexer and searcher nodes. For production workloads, a Kubernetes cluster with the official Helm chart is the recommended path, handling node scheduling, health checks, and rolling updates. Kafka-backed indexing is required to achieve high-availability indexing—without Kafka, indexer restarts will result in brief gaps in ingestion. You are responsible for managing storage lifecycle policies, metastore backups, node scaling, and Quickwit version upgrades, which involve reviewing changelogs and occasionally migrating index configurations.

Quickwit does not currently offer a commercially managed cloud service or a paid support tier with SLAs—the primary support channels are the project’s GitHub issues, Discord community, and documentation. Compared to managed alternatives like Datadog, Elastic Cloud, or Grafana Cloud, self-hosting Quickwit trades per-seat or per-GB pricing for infrastructure management overhead and the absence of a support contract. For teams comfortable operating distributed Rust services on Kubernetes and willing to engage with the open-source community for support, the cost savings on log and trace storage can be dramatic—the project documentation cites estimates of up to 10x cost reduction compared to equivalent Elasticsearch deployments.