binaryornot

BinaryOrNot is a small, pure-Python library that answers one question well: is this file binary or text? It layers three checks — a curated list of 131 known binary extensions, 55 magic-byte file signatures, and a scikit-learn-trained decision tree evaluated over 24 byte-level features (entropy, null/control-byte ratios, BOM markers, and decodability under UTF-8/UTF-16/UTF-32 plus CJK encodings like GB2312, Big5, and Shift-JIS) — to classify files correctly in cases that trip up naive null-byte heuristics, such as UTF-16 text files or CJK-encoded documents with lots of high-ASCII bytes.

Originally built for the cookiecutter project templating tool, it now ships as a standalone package with a single public function, is_binary(), an in-memory variant is_binary_string(), and a small CLI. It has zero runtime dependencies — the decision tree is generated offline via scikit-learn and committed as plain Python — and supports Python 3.10 through 3.14.

What You Get

• A single is_binary(filename) function returning True or False
• An is_binary_string(bytes) function for classifying a chunk you’ve already read into memory
• 131 recognized binary file extensions and 55 magic-byte signatures bundled as CSV data files
• A trained decision tree classifier with no scikit-learn or other ML runtime dependency required to use it
• A binaryornot command-line tool for ad hoc checks from the shell
• A typed public API (py.typed) verified with Astral’s ty type checker in CI

Common Use Cases

• Filtering binary blobs out of full-text search or grep-style indexing
• Deciding whether a diff/version-control tool should render a text diff or treat a file as opaque
• Checking uploaded files in an ingestion pipeline before running text-processing steps
• Skipping template substitution on binary assets in project scaffolding tools like cookiecutter

Under The Hood

Architecture The package is a tight three-module pipeline: check.py exposes the public is_binary() function and CLI entry point, helpers.py does the actual classification work, and tree.py holds an auto-generated decision tree. is_binary() first calls helpers.has_binary_extension(), a cheap extension lookup against a frozenset loaded from data/binary_extensions.csv; if that misses, it reads the first 512 bytes via get_starting_chunk() and hands them to is_binary_string(), which checks the chunk against 55 magic-byte signatures loaded from data/binary_formats.csv before computing 24 numeric features and running them through tree.is_binary(), a nested if/else classifier with no ML runtime involved at inference time. Data tables are loaded once via importlib.resources.files() into module-level frozensets/tuples, so no extra I/O happens per call beyond reading the target file itself.

Tech Stack Pure Python 3.10-3.14, zero runtime dependencies (dependencies = [] in pyproject.toml), built with the hatchling backend and distributed via PyPI with uv used for dependency management. Dev tooling is entirely uv dependency-groups: ruff for linting/formatting, ty (Astral’s newer type checker) for type checking, and pytest/coverage/hypothesis for testing. Documentation is built with zensical and mkdocstrings-python and deployed to GitHub Pages.

Code Quality The test suite spans test_check.py, a dedicated test_encoding_coverage.py that parametrizes tests from the same CSV coverage lists used at runtime (37 text encodings, 49 binary formats per the README), test_encoding_warning.py, and test_sdist.py, which verifies the packaged sdist actually includes the binary test fixtures it needs. hypothesis is a test dependency, indicating property-based test coverage alongside example-based tests. CI runs the full matrix across five Python versions, enforces ruff format/ruff check, runs ty check, and reports branch coverage; the repo also runs CodeQL security analysis and zizmor to scan GitHub Actions workflows themselves for security issues. The public API ships a py.typed marker.

What Makes It Unique Instead of the common null-byte or python-magic-style heuristics, BinaryOrNot trains an explicit decision tree offline (via scripts/train_detector.py using scikit-learn) over engineered byte-level features and ships the trained tree as committed, dependency-free Python code — so consumers get a machine-learned classifier without ever needing scikit-learn installed. Combining that with a magic-signature fast path and explicit CJK/BOM-aware decode checks specifically targets known failure modes of simpler binary-detection heuristics, such as UTF-16 text files full of null bytes being misclassified as binary.