Unidecode

Unidecode is a Python port of Perl’s Text::Unidecode that converts Unicode strings into their closest ASCII approximation using hand-tuned character mapping tables, rather than simply stripping accents. It’s commonly used to generate URL slugs, ASCII identifiers, or displayable text for systems that can’t handle full Unicode.

What You Get

• unidecode() for general-purpose Unicode-to-ASCII transliteration
• unidecode_expect_ascii() and unidecode_expect_nonascii() performance-tuned variants for known input distributions
• Configurable errors handling (ignore/strict/replace/preserve) for characters without a mapping
• A command-line unidecode utility for transliterating stdin, arguments, or files

Common Use Cases

• Generating URL slugs from article or product titles containing non-ASCII characters
• Producing ASCII-safe machine identifiers or filenames from human-readable Unicode strings
• Displaying approximate readable text in legacy systems or terminals without Unicode support
• Normalizing non-Roman names for ASCII-only form fields or database columns

Under The Hood

Architecture - unidecode/__init__.py implements the public unidecode()/unidecode_expect_ascii()/unidecode_expect_nonascii() functions and dispatches each character to one of roughly 180 per-code-block table modules (x000.py through x1d7.py etc.), each a flat list of ASCII replacement strings indexed by codepoint offset — this table-per-block layout keeps lookups O(1) while letting each block be regenerated independently from the underlying Text::Unidecode data. Tech Stack - Pure Python, zero runtime dependencies, Python 3.7+, requiring a ‘wide’ Unicode (UCS-4) build for characters outside the Basic Multilingual Plane; perl2python.pl and tools/check_character_names.py support regenerating tables from the original Perl module’s data. Code Quality - Tested via test_unidecode.py (core transliteration behavior), test_utility.py (the CLI), and test_readme.py (validates the README’s own doctested examples stay accurate), with tox.ini driving multi-version test runs; the project explicitly documents known limitations (CJK transliteration quality, German umlaut handling) as intentional, non-bug behavior in its README FAQ. API Design - The primary entry point is a single pure function taking a string and an errors policy, which is about as low-friction as an API gets; the two ‘expect_ascii’/‘expect_nonascii’ performance variants add a small amount of surface for callers who know their input distribution and want to optimize hot paths.