pyicumessageformat

pyicumessageformat is a small, dependency-free Python3 library that parses ICU MessageFormat message strings — the plural, selectordinal, select, and offset syntax used by translation catalogs from tools like Crowdin, Transifex, and FormatJS — into a plain nested list/dict AST. It is explicitly a from-scratch reimplementation of the JavaScript library format-message-parse, ported so that Python backends can share the same message grammar and AST shape as JavaScript frontends without shelling out to Node.

The library deliberately stops at parsing: it hands back structured data (placeholder names, types, formats, sub-message options, and optional offsets) and leaves interpolation, pluralization-rule lookup, and rendering entirely to the caller. It also supports an optional flat token-list output mode that captures every syntax element with source positions, and a configurable rudimentary XML-style tag parser (with strict, prefix-restricted, or self-closing tag modes) for rich-text placeholders like <b>bold</b> embedded inside translated strings.

What You Get

• A single Parser class with one public method, parse(input, tokens=None), that returns a nested list/dict AST
• AST nodes that distinguish plain text strings from placeholder dicts carrying name, type, format, options, offset, and hash fields
• An optional flat token list (passed in as an empty list) capturing every syntax element — text, syntax, name, type, style, selector, space, offset, number, hash — with text that reconstructs the exact source string
• A configurable options object covering submessage_types, subnumeric_types, maximum_depth, require_other, and the XML-style tag settings (allow_tags, strict_tags, tag_prefix)
• Source position tracking via include_indices, giving start/end offsets on every placeholder and tag node

Common Use Cases

• Server-side rendering of translator-authored ICU MessageFormat strings pulled from translation catalogs into localized, pluralized output
• Building a custom i18n rendering engine that needs the raw AST or token stream rather than a fixed rendering behavior
• Embedding lightweight rich-text tags (like <b> or <link>) inside pluralized chat/bot message templates
• Keeping Python backends aligned with a JavaScript frontend that already uses format-message-parse, by reusing the same grammar and AST shape server-side

Under The Hood

Architecture The library is built around a single Parser class implementing a hand-rolled recursive-descent parser: _parseAST walks the input calling _parsePlaceholder, _parseTag, and _parseText in a loop, threading a single mutable context dict (holding the source string, current index i, length, recursion depth, and the optional tokens accumulator) through every private method rather than returning updated state. constants.py isolates every grammar literal ({, }, <, /, ,, #, the escape character, tag markers) from parser.py, which holds all parsing logic — a clean but very flat separation, with no distinct layers between tokenizing and AST-building; the two happen in the same pass. Because the public surface is a single parse() call, nothing external breaks if the internal recursive-descent implementation changes.

Tech Stack Pure Python3 (python_requires >= 3.6) with zero runtime dependencies. Packaging uses a pyproject.toml declaring setuptools/wheel as the build backend alongside a classic setup.cfg for metadata, and dev-requirements.txt pins only pytest for development. There is no async code, no C extensions, and no bundler — it is distributed to PyPI as a plain sdist/wheel.

Code Quality A real pytest suite in test/test_grammar.py exercises the grammar extensively — over 30 test functions covering plain text, named/positional placeholders, plural/selectordinal/select with offsets, nested sub-messages, all three tag-strictness modes, escaping rules, and a deliberate recursion-limit test that runs 1000 nested sub-messages to confirm RecursionError is rethrown as SyntaxError. Error handling is explicit and intentional: IndexError and RecursionError are caught in parse() and re-raised as SyntaxError with position-annotated messages. Type hints appear only on a handful of small helper functions (isAlpha, isDigit, isSpace) and are absent from the Parser methods themselves. No CI configuration (no .github/workflows) or linter/formatter config exists in the repo, so the test suite isn’t automatically enforced on changes.

What Makes It Unique The library’s main distinguishing choice is being an explicit, faithful Python port of a specific JavaScript grammar (format-message-parse) rather than a novel ICU MessageFormat implementation — its value is giving Python backends the same parsing behavior and AST shape as that JS library, plus a few extra configuration knobs (tag prefixes, per-type require_other rules, offset/hash handling) that the upstream library exposes. It is not attempting a new parsing technique or algorithm; it trades novelty for grammar-compatibility with an existing, known JS parser.