Partial JSON Parser

Partial JSON Parser exists to solve one specific problem: an LLM streaming a JSON response token-by-token produces a string that is valid JSON only once the final token arrives, but applications want to render or act on the data before that happens. The library takes a truncated JSON string plus a set of per-type permission flags and returns a best-effort completion — closing open strings, arrays, and objects — that can then be handed to the standard json.loads (or any parser callable) to get a real Python value back.

Under the hood it ships two independent completion strategies: a recursive-descent parser that walks the string character by character handling every JSON grammar production explicitly, and a faster single-pass structural scanner that tracks bracket/quote positions and short-circuits to O(1) truncation for the common case. Both are exposed through the same small public API (loads/parse_json, ensure_json, fix), and callers get fine-grained control over what counts as an acceptable partial value via the Allow bitflag enum — independently allowing or forbidding partial strings, numbers, arrays, objects, and JSON literals like null/true/NaN/Infinity. It has no runtime dependencies and works back to Python 3.6.

What You Get

• A loads/parse_json function that mirrors the standard library’s json.loads signature but tolerates a truncated input string
• An ensure_json helper that returns the completed JSON string itself, for callers who want the raw text rather than a parsed value
• A low-level fix function that returns the valid slice of the input and the trailing completion suffix separately, useful for debugging or custom rendering
• The Allow IntFlag enum (STR, NUM, ARR, OBJ, NULL, BOOL, NAN, INFINITY, _INFINITY, plus composite groups ATOM/COLLECTION/ALL) for precise per-type partial-tolerance control
• A two-tier exception hierarchy (PartialJSON vs MalformedJSON, both JSONDecodeError subclasses) so callers can distinguish “needs more tokens” from “this is actually broken”
• An optional json-playground CLI (installed via the [playground] extra) for interactively trying the parser against sample partial strings

Common Use Cases

• Rendering an LLM’s structured JSON output live in a chat UI as tokens stream in, instead of waiting for the full response
• Building agent frameworks that need to inspect a tool call’s arguments before the model has finished emitting them
• Recovering usable data from a JSON response that was truncated by a network timeout, token limit, or rate-limit cutoff
• Powering interactive demos or playgrounds that show incremental JSON completion as a teaching tool

Under The Hood

Architecture The core module structure separates a correctness-focused reference implementation from a performance-focused fast path. complete.py implements a recursive-descent parser (fix/_fix calling into mutually recursive complete_any/complete_str/complete_arr/complete_obj/complete_num functions) that walks the string character by character and explicitly handles every JSON value type. myelin.py’s fix_fast takes a different approach: a single regex pass (scan) locates every quote and bracket token, a stack tracks open containers, and the function short-circuits to O(1) string-slicing to truncate back to the last known-good boundary, falling back to the recursive _fix only to resolve whatever value is still open at the tail. api.py’s parse_json/ensure_json are thin façades that pick between the two strategies via a use_fast_fix flag and then hand the completed string to a swappable parser callback (default json.loads). An Allow IntFlag threaded through every function controls which value types may be left partial, and a two-level exception hierarchy in exceptions.py lets callers distinguish incomplete input from malformed input. Changing the Allow abstraction would require touching every completion function, since each one branches directly on flag membership.

Tech Stack Pure Python 3.6+ with zero runtime dependencies — pyproject.toml declares no dependencies at all, only an optional [playground] extra pulling in rich, and a dev dependency group (hypothesis, tqdm, pytest) for testing. It’s packaged with pdm-backend, with its version sourced dynamically from src/partial_json_parser/version.py. A src/overrides.py script runs as a pre/post-build hook to rewrite type-hint syntax for different Python version targets (3.6 pre-build, 3.8 post-build), letting one codebase ship compatible bytecode across supported Python versions. Formatting is enforced via black (160-character line length) and isort.

Code Quality Tests live under tests/ and combine pytest with property-based testing via Hypothesis (test_hypotheses.py, test_examples.py), plus a dedicated test_performance.py benchmark, all wired into composed pdm script tasks. test_examples.py exercises the public API directly with explicit positive and negative assertions, including pytest.raises(PartialJSON) for boundary conditions. Error handling is explicit and typed rather than generic: internal AssertionError/IndexError cases are deliberately caught and re-raised as the library’s own MalformedJSON exception. Naming is concise and consistent across the complete_str/complete_arr/complete_obj/complete_num family, and the codebase carries inline type hints throughout with a pyright configuration, though no CI workflow is visible in a shallow clone and there’s no explicit coverage tooling.

What Makes It Unique The standout design choice is fix_fast’s dual-path strategy: rather than only offering one completion algorithm, the library pairs a correctness-first recursive-descent parser with a structural fast-scan path that falls back to the slow path only for the currently-open tail value. Most comparable partial-JSON-repair libraries implement a single strategy. Combined with the fine-grained Allow bitflag system — independently permitting or forbidding partial strings, numbers, arrays, objects, and each JSON literal — callers get materially more precise control over what “partial” means per value type than the single global partial-tolerance policy typical alternatives offer.