slatedb

SlateDB is a Rust embedded key-value store built as a log-structured merge-tree (LSM-tree), but unlike traditional embedded engines such as RocksDB or LevelDB, it writes its SSTs and write-ahead log directly to object storage — S3, GCS, Azure Blob Storage, MinIO, and Tigris — instead of local disk. That trade lets applications get bottomless storage capacity, built-in durability, and easy replication without operating a dedicated storage tier, at the cost of higher per-operation latency than local-disk engines.

To keep that latency manageable, SlateDB batches writes into periodically flushed memtables, layers in-memory and local-disk block caches, bloom filters, and compression on the read path, and runs background compaction to keep SSTs organized. It supports async batch writes with configurable durability, snapshot reads, MVCC transactions with pluggable isolation levels, change data capture, and database clone/split/merge — all validated by a dedicated deterministic-simulation-testing crate that exercises the production code paths under injected clock and I/O faults. Official language bindings exist for Go, Java, Python, and Node.js/TypeScript, and it’s used in production by companies including Prisma, Dropbox, and several streaming and data-infrastructure vendors.

What You Get

• Object-storage-native LSM engine - SSTs and WAL are written to S3, GCS, Azure Blob Storage, MinIO, or Tigris instead of local disk, giving bottomless capacity and durability without managing a storage tier.
• Async batch writes with configurable durability - WriteHandle.await_durable() or db.flush() let you choose between low-latency acknowledged writes and full durability.
• MVCC transactions and snapshots - DbTransaction and DbSnapshot provide configurable isolation levels for consistent multi-key reads and writes.
• Multi-language bindings - official Go, Java, Python, and Node.js/TypeScript bindings built on the same Rust core via UniFFI and PyO3.
• Change data capture and clone/split/merge - track row-level changes and clone or split a database via manifest projection without copying data.

Common Use Cases

• Embedded state store for stream processors - stateful streaming systems (e.g. Volga, Malstrom) use SlateDB as durable local/cloud-backed state without running a separate database cluster.
• Building custom cloud-native databases - projects like Embucket and HelixDB build higher-level database engines on top of SlateDB’s storage layer.
• Serverless and ephemeral compute state - object-storage-backed persistence suits environments where local disk isn’t durable across restarts.
• Low-ops key-value storage for internal tools - teams that want durable KV storage without provisioning and operating Postgres/Redis/RocksDB clusters.

Under The Hood

Architecture A Db handle (db.rs) orchestrates async writes through a WriteHandle into an in-memory WAL and memtable, with background tokio tasks periodically flushing memtables to SSTs (memtable_flusher/), running size-tiered compaction (compactor.rs, compactor_state.rs, subcompaction.rs), managing object-store I/O and local disk caching (tablestore.rs, cached_object_store/), and coordinating versioned manifests (manifest/store.rs, manifest/invariants.rs) through an optimistic-concurrency oracle (oracle.rs) and a transaction manager (transaction_manager.rs, db_transaction.rs) for isolation-level transactions. The workspace is split into multiple crates (slatedb, slatedb-common, slatedb-txn-obj, slatedb-cli, slatedb-dst, slatedb-bencher) separating the write path, read path (block cache, bloom filters, merge iterator across memtables and SSTs), and background maintenance (compaction, garbage collection via garbage_collector.rs). A dedicated slatedb-dst crate implements deterministic simulation testing — actors, a clocked object store, and a fault-injecting object store that drive the real production code paths under controlled clock skew and I/O failures, rather than relying on flaky integration tests.

Tech Stack Rust 2021 edition on the tokio async runtime (multi-thread), using the object_store crate (with aws/azure/gcp feature flags) for storage backends, flatbuffers for the on-disk SST/manifest binary format, foyer and moka as optional block-cache backends, parking_lot for synchronization, figment for layered configuration, clap for the debugging CLI (slatedb-cli), and pyo3/pyo3-async-runtimes/uniffi for the Python, Go, Java, and Node.js language bindings under bindings/. Errors are typed via thiserror, and benchmarking uses criterion and pprof. It ships as a linkable Rust crate plus generated per-language bindings rather than a standalone server process.

Code Quality Testing is extensive — over a thousand #[test]/#[tokio::test]/#[rstest] occurrences across src/ and tests/, plus proptest property-based tests and the slatedb-dst deterministic-simulation crate, which CI runs hourly (dst-hourly.yaml). Error handling is fully typed (error.rs defines Error/ErrorKind), and clippy.toml enforces project-specific disallowed-methods/disallowed-types lints that ban raw std::time, unscoped RNGs, .unwrap(), and tokio::spawn_blocking in favor of deterministic wrappers (DbRand, SystemClock) — an unusually rigorous determinism-safety discipline for a systems library. CI runs a full matrix (pr.yaml, nightly.yaml, java.yaml, node.yaml, python.yaml, release.yaml) covering the core crate and every language binding, and module boundaries follow consistent, domain-driven naming (batch, compactor, manifest, wal, format).

What Makes It Unique SlateDB’s core bet is writing an LSM-tree’s SSTs and WAL directly to object storage instead of local disk, trading higher per-operation latency for bottomless capacity, built-in durability and replication, and no local-disk operational burden — a distinct niche from RocksDB/LevelDB-style embedded engines. On top of standard LSM techniques (block caching, bloom filters, compression, size-tiered compaction) it adds features uncommon in embedded stores: MVCC transactions with configurable isolation, change data capture, and database clone/split/merge via manifest projection. Its deterministic-simulation-testing harness for validating concurrency and fault behavior under controlled conditions is itself a notable engineering choice, borrowed from FoundationDB-style testing philosophy and rare among Rust storage-engine crates of comparable size.