@sanity/descriptors

@sanity/descriptors is a small TypeScript library for serializing definitions — most notably schema types — into a content-addressable format so they can be shared, cached, and referenced consistently across services. Each descriptor is a JSON object carrying an id derived by recursively hashing its fields with SHA-256, encoded with multibase/multihash conventions, so two descriptors with identical contents always produce the identical id, and descriptors can reference each other recursively by id to form a graph (for example, a schema registry pointing at individual named types).

Beyond encoding, the package implements an efficient synchronization protocol built on an invertible-Bloom-filter-style “set sketch” data structure, based on the paper “Better Space-Time-Robustness Trade-Offs for Set Reconciliation.” This lets a client and server reconcile which descriptors differ between two versions of a set — potentially hundreds of types — by exchanging a compact sketch rather than the full list, and without the client needing to track server-side state between synchronizations.

The library is maintained by Sanity (sanity-io) as infrastructure for Sanity Studio’s schema-to-Content-Lake pipeline, but the encoding and synchronization primitives are generic enough to reuse anywhere a content-addressed, recursively-linked, efficiently-synchronizable data format is useful.

What You Get

• Deterministic descriptor IDs - encode(type, props) produces a stable SHA-256-based id for any JSON-serializable object, independent of key order or which language produced it.
• Recursive descriptor graphs - descriptors can embed other descriptors’ ids, letting a single root id represent an entire linked graph (e.g. a schema registry referencing individual type descriptors).
• Rewrite-map support - the encoder accepts a rewriteMap so values can be substituted mid-encode, useful for normalizing or deduplicating nested structures before hashing.
• Set-based synchronization primitives - SetBuilder, SetSketch, and processSetSynchronization implement an invertible-Bloom-filter-based protocol for reconciling set differences in a small number of round trips.
• Base64url multibase/multihash helpers - encodeBase64, decodeBase64, and encodeBase64Sha256 for working with the same ID encoding used internally.
• Zero runtime dependencies beyond a single SHA-256 implementation (sha256-uint8array), keeping the package small and portable to browser and server environments alike.

Common Use Cases

• Schema registries - encoding CMS/Studio schema type definitions so they can be uploaded once, referenced by id, and deduplicated across projects.
• Content-addressed caching - using descriptor ids as cache keys, since identical content always yields the same id without needing an external UUID or timestamp.
• Incremental schema sync - synchronizing large sets of definitions (hundreds of types) between a client and server by exchanging compact set sketches instead of full payloads.
• Cross-language interoperability - any service that implements the same recursive-hash algorithm can produce or verify the identical id for a given JSON structure, without depending on this specific TypeScript implementation.

Under The Hood

Architecture The package is organized around two largely independent modules that share only the Encoded/ID types: encoder.ts implements the canonical hashing algorithm (an IDEncoder class that walks a JSON value emitting typed byte tags — NULL/TRUE/FALSE/STRING/ARRAY/OBJECT — into a running SHA-256 stream, recursing per-field so object hashing is order-independent by sorting per-field digests before combining them), while reconciler.ts and set.ts implement the synchronization layer (SetSketch, an XOR-based invertible Bloom filter over fixed-size buckets, wrapped by SetBuilder/processSetSynchronization which drive a request/response reconciliation loop against a hypothetical server). sync.ts defines the wire-level request/result types consumed by that loop. Nothing in the core encoding path touches network or storage — callers own the transport, keeping the library usable in both browser and server contexts.

Tech Stack Written in TypeScript, built with @sanity/pkg-utils (pkg build) into dual ESM/CJS output with generated .d.ts types, and tested with Vitest. The only runtime dependency is sha256-uint8array for the hashing primitive; everything else (linting via ESLint + eslint-plugin-simple-import-sort, formatting via Prettier, release automation via semantic-release) is a dev-time concern. A playground/ directory hosts a small demo app exercising the synchronization protocol interactively.

Code Quality The core modules (encoder.ts, reconciler.ts, set.ts) each have a co-located .test.ts file exercising the encode/decode round trip, sketch toggle/decode behavior, and set-builder synchronization flow via Vitest — a reasonable spread for a library this size, though the test suite is compact rather than exhaustive (no fuzz/property testing of the hashing algorithm was found). Types are precise and exported explicitly (EncodableValue, EncodableObject, Encoded<Type, Props>), error handling favors throwing on invariant violations (e.g. numBuckets bounds, non-32-bit numbers) rather than silent fallback, and CI runs the test suite plus a full build across multiple Node LTS versions on every push and PR.

What Makes It Unique Most content-addressing libraries stop at hashing; this one pairs the hash with a genuinely research-backed synchronization protocol (citing a 2024 ICALP paper on set reconciliation) so that large sets of interlinked descriptors can be diffed efficiently without either party holding prior state about the other — a specific, unusual combination aimed squarely at Sanity’s Content Lake sync problem but generic enough to apply anywhere content-addressed graphs need cheap incremental reconciliation.