jsonlines

jsonlines simplifies working with the JSON Lines (also known as ndjson) data format, where each line of a file or stream is a standalone, valid JSON value. Rather than every project reinventing str/bytes handling, UTF-8 BOM detection, and line-based error reporting, jsonlines wraps it all in a small, well-tested Reader/Writer API.

The library exposes a convenience jsonlines.open() function for the common case of reading or writing a file, plus Reader and Writer classes that can wrap any file-like object or iterable of lines for more advanced streaming use. It transparently uses orjson or ujson for faster decoding when installed, falling back to the standard library json module otherwise, while keeping encoding on the stdlib json module by default for predictable output.

What You Get

• A jsonlines.open() convenience function that returns a Reader or Writer based on file mode, usable as a context manager
• A Reader class that can wrap a file object or any iterable of lines, with .read() and .iter() methods supporting type validation and allow_none
• A Writer class supporting compact output, sorted keys for deterministic output, custom dumps callables, and per-write flushing
• Automatic use of orjson or ujson for faster JSON decoding when either is installed, with a transparent fallback to the standard library
• Descriptive InvalidLineError exceptions carrying the offending line and line number for easier debugging of malformed data

Common Use Cases

• Streaming large datasets line-by-line without loading an entire JSON array into memory
• Reading and writing structured log files or event streams in the ndjson format
• Exchanging data between data-processing pipelines (e.g. ML training data, ETL jobs) that use JSON Lines as an interchange format
• Appending records incrementally to a growing file using write mode ‘a’ without re-serializing existing data

Under The Hood

Architecture: jsonlines is built around two small classes, Reader and Writer, both subclassing a shared ReaderWriterBase (in jsonlines/jsonlines.py) that handles context-manager semantics and lazy file closing. Reader wraps any iterable yielding str/bytes lines and lazily decodes each with a pluggable loads callable, tracking line numbers via enumerate() for error reporting. Writer wraps a writable file-like object, auto-detects whether it accepts str or bytes on __attrs_post_init__, and encodes via a pluggable dumps callable, defaulting to a configured json.JSONEncoder.encode. The top-level open() function is the primary entry point, opening a file in the right mode/encoding and instantiating the correct class.

Tech Stack: Pure Python 3.8+, with a single runtime dependency on attrs (>=19.2.0) for class boilerplate (@attr.s(auto_attribs=True)). It optionally detects orjson and ujson at import time via try/except imports to accelerate decoding, but never adds them as hard dependencies. Packaging uses a plain setup.py/setup.cfg (setuptools), not a pyproject.toml-based backend.

Code Quality: The test suite (tests/test_jsonlines.py, 313 lines, and tests/test_typing.py, 95 lines) exercises reading, writing, error handling, and BOM/type-validation edge cases. Typing is thorough — the public API uses extensive @overload signatures on Reader.read()/.iter() and the module ships a py.typed marker for downstream type checkers, with mypy.ini configured for strict checking. Error handling is explicit via a dedicated Error/InvalidLineError exception hierarchy rather than silent failures.

API Design: The API is intentionally small and idiomatic: jsonlines.open(path, mode) mirrors the builtin open() signature, both Reader and Writer support the with statement, and Reader is directly iterable. Type-checked reads (type=dict, allow_none=True) and skip_invalid/skip_empty flags cover common streaming needs without extra ceremony, keeping boilerplate to a few lines for the common case.