gcloud-aio-bigquery

gcloud-aio-bigquery is an async Python client for the Google Cloud BigQuery REST API, built on aiohttp so BigQuery calls no longer block your event loop. It wraps jobs, tables, and datasets in small, task-focused classes that talk directly to BigQuery’s v2 REST endpoints, handle OAuth token refresh via a shared gcloud-aio-auth Token, and normalize BigQuery’s nested field-and-value response format back into ordinary Python values.

The package is generated from the same codebase as gcloud-rest-bigquery, a threadsafe requests-based twin published under a different PyPI name, so teams choosing between asyncio and synchronous code get identical APIs. It supports streaming inserts, load jobs from Cloud Storage, query jobs (sync and async), table copies, and dataset management, plus a local emulator mode via BIGQUERY_EMULATOR_HOST for testing without hitting live GCP.

What You Get

• Table class - create, patch, delete, and stream-insert rows into BigQuery tables via insertAll, with configurable per-row insert IDs for dedup control.
• Job class - run synchronous queries, insert async query/load/copy jobs, poll job status, and cancel or delete running jobs against BigQuery’s /jobs endpoints.
• Dataset class - list, get, create, and delete datasets, and list the tables within them.
• query_response_to_dict utility - converts BigQuery’s nested {'f': [{'v': ...}]} row format into typed, ordinary Python dictionaries, handling NULLABLE, REPEATED, and RECORD fields.
• Emulator support - point BIGQUERY_EMULATOR_HOST at a local BigQuery emulator to test without live GCP credentials or costs.
• Shared auth via gcloud-aio-auth - reuses a single Token/session across bigquery, storage, and other gcloud-aio-* clients instead of re-authenticating per package.

Common Use Cases

• Streaming event data into BigQuery - async workers batch and insert rows into a table in real time using Table.insert.
• Scheduled ETL/query jobs - a service kicks off a BigQuery query job via Job.insert_via_query, polls for completion, and reads results with query_response_to_dict.
• Loading GCS exports into BigQuery - Table.insert_via_load loads CSV, Avro, Parquet, or JSON files staged in Cloud Storage directly into a destination table.
• Table-to-table copies - Table.insert_via_copy duplicates or archives a table into another project or dataset without downloading data client-side.
• Local integration testing - CI pipelines run against the BigQuery emulator instead of a live project, using the same client code as production.

Under The Hood

Architecture gcloud-aio-bigquery follows a thin, REST-first layered design: a shared BigqueryBase class in bigquery.py owns HTTP session handling (via gcloud-aio-auth’s AioSession), OAuth token acquisition, and the low-level _post_json/_get_url/_delete verbs, while Job, Table, and Dataset each subclass it to expose resource-specific methods that build the appropriate REST payload and call those verbs against BigQuery’s /v2 endpoints. Job-creating methods on Table (insert_via_load, insert_via_copy, insert_via_query) return a fresh Job instance wired to the same session and token, letting callers immediately poll or cancel the job they just submitted without re-authenticating. Response normalization is isolated in utils.py’s flatten/parse functions, which recursively walk BigQuery’s {'f': [{'v': ...}]} response shape into plain Python values based on the query’s returned schema. Because a BUILD_GCLOUD_REST flag is checked at import time to swap aiohttp for requests, the same source tree compiles into both the async gcloud-aio-bigquery and sync gcloud-rest-bigquery packages, so any change to a job/table/dataset method must remain compatible with both session types.

Tech Stack The library targets Python 3.10+ and is built with Poetry (pyproject.toml declares poetry-core as the build backend), depending on gcloud-aio-auth (>=3.1.0,<6.0.0) for session and token management and, transitively, aiohttp for async HTTP in the -aio variant (requests for the -rest variant, built from the same source via a separate pyproject.rest.toml). It’s a pure REST API client targeting BigQuery’s v2 REST surface, not gRPC or the official google-cloud-bigquery SDK, and has no ORM, web framework, or database driver dependency. Dev dependencies include pytest, pytest-asyncio, pytest-mock, and tenacity for retry-oriented tests, plus local path dependencies on sibling gcloud-aio-auth/datastore/storage packages from the monorepo for integration testing. CI runs on CircleCI, and Renovate keeps dependencies current.

Code Quality Unit tests exist for both the request-building helpers (bigquery_test.py exercises Table._make_insert_body) and the response-parsing utilities (utils_test.py covers flatten/parse across BigQuery’s field types and modes), plus a separate integration suite (tests/integration/smoke_test.py) meant to run against a live or emulated BigQuery project. Error handling is mostly implicit — HTTP errors surface as whatever gcloud-aio-auth’s session layer raises, and Job.result() raises a plain Exception on job errors and OSError while a job is still pending, rather than typed exception classes. The codebase is fully type-hinted (py.typed markers throughout, modern union syntax like str | None) and checked with mypy per the repo-root mypy.ini; pylint suppressions are used selectively rather than broadly.

API Design The API is intentionally minimal and closely mirrors BigQuery’s REST resource model (Job, Table, Dataset) rather than wrapping it in a heavier ORM-like abstraction, which keeps the learning curve low for anyone who already knows the BigQuery REST reference — each method links directly to its Google Cloud docs page in a comment. The standout ergonomic choice is that job-creating Table methods return a ready-to-use Job object instead of a raw job ID, so polling or cancelling a just-submitted job needs no extra boilerplate. Sharing one codebase to produce both an asyncio and a synchronous (gcloud-rest) package via a single build flag is a pragmatic but not particularly novel technique, also used across the other gcloud-aio-* siblings. There’s no query builder, schema validation, or DataFrame integration — callers write raw BigQuery SQL and handle typed results themselves via query_response_to_dict.