iceberg

iceberg is the core crate of Apache Iceberg Rust, a native Rust implementation of the Apache Iceberg open table format for huge analytic datasets. It gives Rust programs first-class access to Iceberg tables: reading and writing data files, managing table metadata and snapshots, evolving schemas and partitions, and scanning tables with predicate pushdown into Apache Arrow record batches.

Governed by the Apache Software Foundation, the library implements the Iceberg specification directly rather than wrapping the Java implementation, so it can be embedded in Rust-based query engines, ETL tools, and data services. It pairs with catalog crates (REST, Glue, and others) to resolve and commit table state against a shared metastore.

What You Get

• A native Rust implementation of the Iceberg table spec, including metadata, snapshots, and manifests
• Readers and writers that operate on Parquet data files and produce Apache Arrow record batches
• Table scan APIs with column projection and predicate pushdown for efficient queries
• Schema and partition evolution plus transaction support for committing table changes through a catalog

Common Use Cases

• Reading Iceberg tables from a Rust query engine or data-processing service
• Writing and appending data files to Iceberg tables from a Rust ETL pipeline
• Building a lakehouse integration in Rust without depending on the JVM Iceberg implementation
• Managing table metadata, snapshots, and schema evolution programmatically

Under The Hood

Architecture - The repository is a Cargo workspace under crates/ where the iceberg crate holds the core spec implementation (table metadata, schema, partition spec, manifests, snapshots) and companion crates provide catalogs (iceberg-catalog-rest, Glue, and others) and integrations. Data access is abstracted through OpenDAL so tables can live on S3, GCS, local disk, or other object stores. A TableScan planner resolves the relevant data files from manifests and streams their contents as Arrow RecordBatches, applying projection and filter pushdown.

Tech Stack - Almost entirely Rust (98%), built on Apache Arrow for the in-memory columnar model, Parquet for on-disk data, OpenDAL for storage IO, and async Rust throughout. Catalog crates add HTTP (REST) and cloud-provider clients.

Code Quality - This is a high-velocity ASF project (1,400+ commits, 177 contributors, ~74 commits/month) with an extensive test suite, integration tests against real catalogs, CI, and adherence to Apache release governance. The clean split between the core spec crate and pluggable catalog/IO layers keeps responsibilities well isolated.

API Design - The public API models Iceberg concepts faithfully (Catalog, Table, Transaction, TableScan, Schema), so users familiar with Iceberg elsewhere find the surface recognizable. Async, builder-style construction and Arrow-native output make it ergonomic to integrate into query engines, though callers do need to understand Iceberg’s metadata model. The project is pre-1.0, so some API churn across releases is expected.