ijson

ijson is an iterative JSON parser that exposes standard Python iterator interfaces, letting you process JSON documents as a stream rather than loading them entirely into memory. You feed it a file-like object and pull out native Python objects located under a prefix, so multi-gigabyte documents and network streams parse with a near-constant memory footprint.

It offers several interchangeable backends, from a pure-Python implementation to fast C-accelerated backends built on the yajl2 library (via a C extension, CFFI, or ctypes), and a small set of composable generators (basic_parse, parse, items, kvitems) for building custom streaming pipelines.

What You Get

• High-level items() and kvitems() functions that yield native Python objects located under a JSON prefix
• Low-level basic_parse() and parse() generators emitting a stream of parsing events for custom pipelines
• Multiple interchangeable backends: pure Python plus C-accelerated yajl2 (C extension, CFFI, ctypes)
• Both blocking (iterator) and coroutine/async interfaces for feeding data as it arrives
• Prebuilt binary wheels for major platforms and Python versions

Common Use Cases

• Parsing JSON documents far larger than available memory by streaming objects one at a time
• Processing JSON as it streams in from an HTTP response, socket, or subprocess
• Extracting specific nested values from a large payload without deserializing the entire structure

Under The Hood

Architecture - ijson is organized around a small set of composable generators defined in src/ijson/common.py: basic_parse emits primitive parse events, parse enriches them with prefix paths, and items/kvitems build native objects under a chosen prefix. The actual event production is delegated to interchangeable backends in src/ijson/backends/ (python.py for pure Python, plus yajl2, yajl2_c, yajl2_cffi and a ctypes variant wrapping the C yajl library). A backend is selected at import time and adapters.py bridges the blocking iterator and coroutine/async styles over the same core, so the public API stays identical regardless of backend.

Tech Stack - Written in Python with an optional C extension (cextern/ and backends/ext) that links against the yajl2 library for speed; CFFI and ctypes backends provide C acceleration without building the extension. Packaging uses setuptools with a custom setup.py that probes for yajl at build time, and binary wheels are produced with cibuildwheel via GitHub Actions.

Code Quality - The repository carries a thorough pytest suite under tests/ (basic_parse, items, kvitems, generators, dump, adapters, benchmark) that is run against every backend to keep behavior consistent, along with coverage reporting. The design keeps each backend behind a uniform contract, which makes the codebase small per-backend and heavily tested at the boundary.

API Design - The public interface is intentionally minimal and idiomatic: pass a file-like object and a prefix string to items() and iterate. The four-generator model (basic_parse, parse, items, kvitems) composes cleanly for advanced use, and because the same functions exist across backends and in async form, users rarely need to change code to trade portability for speed.