simd-json

simd-json is a Rust implementation of the simdjson design: a JSON parser that leans on SIMD (AVX2/SSE4.2 on x86, NEON on aarch64, simd128 on wasm) to classify structural bytes and validate strings in wide parallel passes instead of a byte-at-a-time scanner. It runs a two-stage pipeline — a structural indexing pass followed by tape construction — and exposes that result through three different consumption models: owned and borrowed DOM Value types for ad-hoc JSON manipulation, a raw Tape API for the lowest-overhead access to parsed structure, and a Serde-compatible from_slice/to_vec layer that lets it act as a near-drop-in accelerator for code already built on serde_json.

Because SIMD intrinsics and some of the buffer tricks used to avoid safe-Rust overhead are inherently unsafe, the crate leans heavily on property-based and fuzz testing (structural JSON generation, string fuzzing, and upstream simdjson/JSONTestSuite corpora) to keep that surface area honest, and it runs a dedicated weekly big-endian (s390x) CI job to catch endianness bugs in its byte-reinterpretation hot paths. Feature flags let consumers opt into behavior trade-offs — deduplicating object keys, treating oversized integers as floats, 128-bit integer support, or swapping the float representation for Eq compatibility — rather than baking one opinion into the parser.

What You Get

• Runtime CPU-feature detection that picks AVX2, SSE4.2, NEON, or wasm simd128 automatically, with a native fallback for unsupported hardware
• Two DOM value types (BorrowedValue, OwnedValue) for working with parsed JSON without knowing its shape ahead of time
• A lower-overhead Tape API for reading parsed structure directly without materializing a full value tree
• A Serde-compatible layer (from_slice, from_str, to_vec, to_writer, and pretty-printing variants) that mirrors serde_json’s function signatures
• Feature flags for 128-bit integers, duplicate-key handling, oversized-integer-as-float coercion, and an Eq-compatible float wrapper

Common Use Cases

• Dropping in as a faster parser behind existing serde_json-based (de)serialization code with minimal API changes
• High-throughput services that parse large volumes of JSON payloads (log ingestion, API gateways, streaming pipelines) where parse time is a measurable cost
• Building tools that need to inspect or transform arbitrary JSON documents via the DOM Value APIs without a fixed schema
• Benchmarking or replacing JSON parsing in performance-sensitive data pipelines where allocator choice (mimalloc/jemalloc/snmalloc) is already being tuned

Under The Hood

Architecture simd-json runs a two-stage pipeline defined in src/stage2.rs and the per-architecture backends under src/impls/ (avx2, sse42, neon, simd128, native, and an experimental portable std::simd backend). The first stage classifies structural bytes (braces, brackets, colons, commas, quotes) and validates strings/atoms (is_valid_true_atom, is_valid_false_atom, is_valid_null_atom in stage2.rs) using wide SIMD compares; the second stage walks that structural index to build either a Tape or a Value tree, mediated by a Deserializer and StackState that track nesting depth against DEFAULT_MAX_DEPTH. src/impls/mod.rs selects the backend module via cfg gates on target architecture and the portable feature, so the same stage-2 logic runs against whichever SIMD backend compiled in for the target. src/value/ splits the DOM representation into borrowed, owned, lazy, and tape submodules, and src/serde/ layers serde::Serializer/Deserializer implementations on top of the same tape so the Serde-facing API doesn’t duplicate the core parser.

Tech Stack The crate targets Rust 2024 edition (rust-version 1.88) and depends on simdutf8 for validated UTF-8 checks, value-trait for shared value-type traits, halfbrown for its hash map implementation, ref-cast for zero-cost newtype casting, and optional ahash/beef/serde/serde_json behind feature flags. Dev dependencies include criterion and proptest for benchmarking and property testing (excluded on wasm targets) and core_affinity/perfcnt for pinned, counter-instrumented perf runs. CI (GitHub Actions) covers stable and nightly toolchains across Linux/macOS/Windows with RUSTFLAGS combinations that disable AVX2/SSE4.2/pclmulqdq to force each fallback path, plus a scheduled QEMU-based s390x (big-endian) job.

Code Quality src/lib.rs opens with #![deny(warnings)] and #![deny(clippy::all, clippy::pedantic, clippy::unwrap_used, clippy::unnecessary_unwrap, missing_docs)], a strict bar for a crate that relies extensively on unsafe. Tests are spread across src/tests.rs, src/*/tests submodules, and a tests/ directory (jsonchecker.rs, depth.rs, alloc.rs, ordered_float.rs, serenity.rs) covering conformance against JSON test suites, allocation behavior, and downstream-crate compatibility; a fuzz/ directory with passing, failing, and real targets extends coverage to malformed and adversarial inputs, run against both upstream simdjson’s and JSONTestSuite’s pass/fail corpora. Error handling is typed through an explicit ErrorType enum (src/error.rs) rather than panics or opaque strings for parser-facing failures, though correctness at the unsafe boundary is enforced by testing discipline rather than by the type system alone.

What Makes It Unique The defining technical choice is porting simdjson’s structural-indexing algorithm into safe-Rust-adjacent code while still exposing three distinct consumption models (DOM, tape, Serde) over one shared parse result, rather than committing to a single output representation the way most JSON crates do. The known-key feature is a good example of the trade-offs made explicit: swapping ahash (DOS-resistant) for fxhash (repeatable, memoizable) is offered as an opt-in for workloads that repeatedly look up a small set of well-known keys, rather than being silently chosen for the user. Combined with the weekly big-endian CI job and the property/fuzz-testing investment around its unsafe core, the crate treats SIMD-parser correctness as a first-class, continuously verified concern rather than a one-time optimization.