chardet

chardet is a pure TypeScript library that determines the most likely character encoding of raw bytes — a Buffer, Uint8Array, or file on disk — by tallying byte-occurrence statistics and running them through a set of dedicated recognisers for UTF-8, UTF-16/32, ISO-2022 (JP/KR/CN), Shift_JIS, Big5, EUC-JP/KR, GB18030, the ISO-8859 family, several Windows code pages, and KOI8-R. It returns either a single best-guess encoding via detect() or a full confidence-ranked list via analyse(), so callers can inspect alternatives when the input is ambiguous.

The library has no runtime dependencies and no native bindings, ships a ~22KB package, and works identically in Node.js and the browser by swapping a small filesystem shim. It is commonly reached for when data of unknown provenance — uploaded files, scraped or imported text, log data from mixed regional sources — needs to be decoded correctly before it can be parsed as UTF-8 text.

What You Get

• A detect() function that returns the single most probable encoding name for a Buffer or Uint8Array
• An analyse() function returning every candidate encoding with a confidence score and, where applicable, a detected language
• detectFile() (async) and detectFileSync() helpers that read from disk and run detection directly against file contents
• sampleSize/offset options to scan only part of a large file for faster, lower-accuracy detection
• Zero runtime dependencies and no native/C++ bindings — a pure TypeScript implementation
• Identical behavior in Node.js and browsers via a swappable filesystem shim

Common Use Cases

• Detecting the encoding of user-uploaded text files before decoding them to UTF-8 in a backend service
• Picking the correct decoder for CSV or data-import pipelines that ingest files from varied regional sources
• Sniffing file encoding in CLI tools (editors, search utilities) to avoid mojibake when rendering contents
• Sampling only the first few KB of very large files or logs to cheaply detect encoding before full processing

Under The Hood

Architecture The library follows a simple, single-purpose strategy pattern. The entry point (src/index.ts) exports detect/analyse/detectFile/detectFileSync, all built on top of a fixed array of Recogniser instances (Utf8, UTF_16BE/LE, UTF_32BE/LE, sjis, big5, euc_jp, euc_kr, gb_18030, ISO_2022_JP/KR/CN, the ISO_8859_x family, several windows_x code pages, KOI8_R, and Ascii), each implementing a shared match/name/language contract defined in src/encoding/index.ts. analyse() builds one shared Context (a 256-slot byte histogram plus C1-byte and raw-buffer flags), maps every recogniser over it, filters out non-matches, and sorts by confidence — no dependency injection, no configuration layer, no external state. The one platform seam is fs/node.ts vs fs/browser.ts, swapped via package.json’s browser field so detectFile/detectFileSync behave identically under Node and bundlers. Adding a new encoding is a pure extension (a new class plus an entry in the recognisers array), which shows the shared interface was deliberately kept minimal to make the recogniser set easy to grow.

Tech Stack 100% TypeScript, targeting both Node.js and the browser from one build. package.json declares zero runtime dependencies; devDependencies are limited to typescript, vitest with @vitest/coverage-v8 for testing, tsx for running scripts, prettier for formatting, and semantic-release for automated versioned publishing. The build step (tsc -p tsconfig.build.json) emits lib/*.js plus .d.ts typings consumed via the main/typings package.json fields. GitHub Actions workflows handle test runs and a semantic-release publish pipeline, and Renovate is configured for automated dependency-update PRs.

Code Quality Every encoding module (ascii.ts, utf8.ts, unicode.ts, mbcs.ts, sbcs.ts, iso2022.ts) has a matching *.test.ts file, alongside index.test.ts and utils.test.ts, run via Vitest with coverage collection and a separate type-check-only test script in CI. Error handling is explicit rather than swallowed: invalid (non-buffer) input throws a descriptive error, and file-reading paths reject on I/O errors while still closing file descriptors. Naming is consistent with established IANA/ICU encoding identifiers, strict TypeScript is used throughout with typed public interfaces (Recogniser, Context, Match, EncodingName), and Prettier enforces formatting. There is no dedicated ESLint configuration, so lint-equivalent checks rely on the TypeScript compiler and formatter alone.

What Makes It Unique The detection heuristics themselves are not novel — they are openly credited, direct ports of the statistical byte-occurrence approach used by ICU4J and uchardet for each encoding family. The genuinely distinctive parts are packaging and ergonomics: a dependency-free, very small (~22KB) dual Node/browser build, a confidence-ranked analyse() result rather than a single opaque guess, and a sampleSize/offset option that lets callers trade completeness for speed on large files — a practical knob that many comparable encoding-detection libraries don’t expose.