gcp-bigquery-client

gcp-bigquery-client is an async Rust library that wraps the full GCP BigQuery REST API surface behind a typed, builder-driven interface. It handles dataset and table lifecycle management, the tabledata streaming-insert API for pushing rows from ordinary Rust structs implementing Serialize, SQL query execution with a cursor-style ResultSet for reading typed columns back out, and partial support for the newer high-throughput BigQuery Storage Write API built on gRPC via tonic and prost.

Authentication goes through yup-oauth2, supporting service account key files, workload identity, the OAuth installed flow, and application default credentials, so the same client works from a local dev machine, CI, or a GCP-hosted workload without code changes. The crate re-exports yup-oauth2 directly so consumers never have to hand-sync a matching version. It is still pre-1.0 and its public API can change, but coverage of dataset, table, tabledata, job, model, project, and routine endpoints is essentially complete, with active maintainers merging community PRs against a growing GitHub discussions backlog.

What You Get

• A single Client struct exposing .dataset(), .table(), .job(), .tabledata(), .routine(), .model(), .project(), and .storage() sub-APIs, each backed by a shared authenticated reqwest::Client
• Builder-pattern constructors for Dataset and Table (labels, expiration, time partitioning, nested TableFieldSchema records) instead of hand-assembled JSON payloads
• A TableDataInsertAllRequest builder plus #[derive(Serialize)]-based row insertion for the BigQuery Streaming API, with a ResultSet cursor for reading typed columns back out of query responses
• Five authentication paths through yup-oauth2 — service account key file, in-memory ServiceAccountKey, GCP workload identity, OAuth installed flow, and application default credentials
• Partial BigQuery Storage Write API support (protobuf-encoded append streams over tonic/gRPC) for higher-throughput ingestion than the REST streaming insert path
• A typed BQError enum (via thiserror) distinguishing auth failures, transport errors, response errors, and column-access mistakes, instead of an opaque catch-all error

Common Use Cases

• Streaming application events or metrics into a BigQuery table from a Rust backend using row structs that already implement Serialize
• Provisioning and tearing down BigQuery datasets and tables as part of an integration test suite or data-pipeline bootstrap step
• Running parameterized SQL queries against BigQuery and reading results back into typed Rust values via ResultSet
• High-throughput ingestion pipelines that need the Storage Write API’s gRPC append-stream path instead of the REST streaming insert endpoint
• Authenticating BigQuery access uniformly across local development, CI, and GCP-hosted deployments via workload identity or application default credentials

Under The Hood

Architecture The crate centers on a single Client struct (src/lib.rs) that owns one shared reqwest::Client plus an Arc<dyn Authenticator>, cloned into eight sub-API structs — DatasetApi, TableApi, JobApi, TableDataApi, RoutineApi, ModelApi, ProjectApi, and StorageApi — each in its own module (dataset.rs, table.rs, job.rs, tabledata.rs, routine.rs, model_api.rs, project.rs, storage.rs) that maps directly onto a BigQuery REST resource. Construction is deferred to ClientBuilder (client_builder.rs), which resolves one of five Authenticator implementations (auth.rs) before assembling the client, so authentication strategy is decided once at startup and every sub-API shares the same authenticated transport and configurable base URL. The storage module is architecturally distinct: it speaks gRPC via tonic/prost against protobuf types generated at build time (build.rs, src/google/) rather than the REST/JSON path the other modules use, and pools write-stream connections with deadpool. Swapping the core reqwest-based transport would require touching all eight sub-APIs; the gRPC storage path is comparatively isolated.

Tech Stack Rust 2021 edition, built on tokio (multi-thread runtime), reqwest (rustls-tls by default, optional native-tls feature) for the REST surface, and tonic/prost/tonic-prost with build-time codegen for the Storage Write API’s gRPC/protobuf layer. Authentication runs through yup-oauth2 12.x (service account, workload identity, installed-flow, ADC support), which the crate re-exports so consumers stay version-locked automatically. Errors are modeled with thiserror, connection pooling for gRPC streams uses deadpool, and dev-dependencies (wiremock, tokio-test, fake, tempfile) support integration-style testing against mocked and live BigQuery endpoints.

Code Quality Tests live inline as #[tokio::test]/#[test] functions across dataset.rs, table.rs, job.rs, tabledata.rs, auth.rs, and storage.rs, and CI (.github/workflows) runs cargo test against a live BigQuery project via injected service-account credentials, alongside separate rustfmt --check and clippy -D warnings jobs plus a scheduled cargo audit security check — a stricter setup than most crates of this size. Error handling is explicit throughout via the typed BQError enum rather than unwrap/panic in library code, and public APIs are documented with doc comments that render on docs.rs.

API Design The builder pattern for Dataset/Table/insert-request construction keeps call sites readable even for deeply nested table schemas, and grouping all endpoints behind client.dataset(), client.table(), etc. gives the API a discoverable, resource-oriented shape that mirrors BigQuery’s own REST hierarchy. Getting started requires only a service account key path and a handful of async calls, though the crate is explicit that its public surface is still pre-1.0 and may change before a stable release.