ftfy

ftfy (“fixes text for you”) repairs text that has been mangled by encoding mistakes — the classic “mojibake” that results when UTF-8 bytes get decoded as Latin-1, Windows-1252, or another mismatched codec, sometimes multiple times in a row. Rather than guessing blindly, it uses a heuristic in ftfy.badness that recognizes the specific character patterns produced by common encoding mix-ups, and only applies a fix when doing so measurably improves the text — a deliberate design choice to avoid corrupting text that was already correct.

Beyond mojibake, ftfy can decode stray HTML entities found outside of HTML, uncurl smart quotes that block correct decoding, strip terminal escape codes, restore dropped non-breaking spaces, and normalize inconsistent line breaks and Unicode ligatures. Every fixer function is individually toggleable through a TextFixerConfig object, and fix_and_explain() returns the exact sequence of decode/encode/apply steps it took, so callers can audit or reproduce a fix rather than trust a black box.

It ships as a small, dependency-light Python package (its only runtime dependency is wcwidth) with a ftfy command-line tool for fixing files directly, and has been widely adopted as a text-cleaning step in NLP and dataset-preparation pipelines.

What You Get

• A single fix_text() call that applies sane default fixes for the vast majority of mojibake and text-glitch cases with no configuration
• A TextFixerConfig object exposing granular, individually-toggleable fixers (HTML unescaping, quote uncurling, control-character removal, line-break normalization, and more)
• fix_and_explain() and fix_encoding_and_explain(), which return a step-by-step trace of exactly which decode/encode/fix operations were applied
• Custom codecs (sloppy encodings, utf-8-variants) registered under ftfy.bad_codecs for decoding byte sequences that standard Python codecs reject
• A ftfy command-line tool for fixing text files or piped stdin/stdout directly, without writing any Python
• Extensive Sphinx documentation covering the detection heuristic, supported encodings, and edge cases in detail

Common Use Cases

• Cleaning up text scraped from the web or pulled from legacy databases where encoding metadata was lost or wrong
• Preprocessing corpora for NLP and machine-learning pipelines, where inconsistent mojibake would otherwise pollute tokenization and embeddings
• Fixing CSV/JSON exports from systems that silently mis-decoded UTF-8 as Windows-1252 or Latin-1 at some point in a data pipeline
• Repairing user-generated content (forum posts, tweets, chat logs) that accumulated multiple layers of double-encoding over time
• One-off cleanup of text files from the command line via the bundled ftfy CLI tool

Under The Hood

Architecture The library is structured as a single top-level module, ftfy/__init__.py, that orchestrates a pipeline of composable “fixer” functions rather than a class hierarchy or plugin system. The public entry points — fix_text(), fix_and_explain(), fix_encoding() — walk text through a configurable sequence of transformations drawn from a FIXERS registry dict, with each step implemented as a standalone function in ftfy/fixes.py. A heuristic in ftfy/badness.py scores how “mojibake-like” a span of text looks and acts as the pipeline’s stopping condition, deciding when re-decoding stops improving the text; encoding-specific character tables live in ftfy/chardata.py, and non-standard codecs are registered separately under ftfy/bad_codecs/. Configuration flows through an immutable TextFixerConfig NamedTuple instead of mutable global state, and a thin ftfy/cli.py wraps the same library functions for file-based use. Because badness.py is the shared stopping condition for every fixer, changes to its heuristic would ripple into every public function’s behavior at once.

Tech Stack ftfy is a pure Python 3.9+ package built with hatchling, declaring a single runtime dependency (wcwidth) for terminal-width-aware CLI output. Development tooling is uv-managed (a committed uv.lock), with pytest for tests, an unusually strict ruff configuration (bugbear, pyflakes, isort, pep8-naming, annotations, pyupgrade, and more rule sets enabled), and mypy for type-checking via a dedicated mypy.ini. Documentation is built with Sphinx and the furo theme and published to Read the Docs. Releases are fully automated through a GitHub Actions workflow that builds, Sigstore-signs, and publishes to PyPI on tagged pushes.

Code Quality The test suite (test_bytes.py, test_characters.py, test_cli.py, test_encodings.py, test_entities.py, plus a JSON-driven example-regression test) exercises encoding edge cases, CLI behavior, and known-good/known-bad text pairs, run through pytest and tox. The codebase is fully type-annotated with modern typing constructs (from __future__ import annotations, TYPE_CHECKING imports, NamedTuple-based configs) and ships a py.typed marker for downstream consumers. However, the only GitHub Actions workflow present handles PyPI release automation — there is no CI workflow that runs the test suite or linter on every push or pull request, so quality enforcement currently relies on local tooling and maintainer discipline rather than an automated gate.

API Design The public surface is deliberately small: a single fix_text(text) call covers the common case with sensible defaults, while TextFixerConfig exposes fine-grained per-fixer toggles for advanced callers without requiring subclassing or a plugin system. fix_and_explain() is a distinctive addition — it returns the exact sequence of decode/encode/transform steps applied as structured ExplanationStep tuples, giving callers a way to audit or reproduce a fix instead of trusting an opaque transformation. Extensive prose documentation (docs/explain.rst, docs/heuristic.rst, docs/detect.rst) walks through the underlying detection heuristic in detail, which is unusually transparent for a “fix broken text” utility.