tldextract

tldextract solves a problem that naive string splitting gets wrong: given a hostname like forums.bbc.co.uk, which part is the “domain”? Splitting on the last dot gives co, but the real registrable domain is bbc.co.uk. tldextract answers this correctly by consulting the community-maintained Public Suffix List (PSL), which enumerates both public suffixes like .com and .co.uk and “private” suffixes like blogspot.com that browsers also treat specially for cookie scoping and security decisions.

The library fetches the PSL over HTTP on first use, caches it to disk (XDG-compliant, keyed per Python environment to avoid cross-venv corruption), and falls back to a bundled snapshot if no network or cache is available. Extraction itself is a reverse-label trie walk that correctly handles PSL wildcard rules, exceptions, and IDNA/punycode-encoded labels. Results come back as an ExtractResult dataclass exposing not just subdomain/domain/suffix but derived properties like fqdn, ipv4/ipv6 detection, and reverse-DNS notation — plus a tldextract CLI for one-off lookups.

What You Get

• A single tldextract.extract() function and TLDExtract class returning an ExtractResult with subdomain, domain, suffix, and private-domain metadata
• Automatic Public Suffix List fetching, disk caching, and offline snapshot fallback so extraction works without network access
• Punycode/IDNA-aware suffix matching so internationalized domains resolve to the correct suffix boundary
• A tldextract command-line tool for quick hostname lookups, with JSON output support
• Derived ExtractResult properties (fqdn, ipv4, ipv6, reverse_domain_name, top_domain_under_public_suffix) that cover common follow-on needs without extra code

Common Use Cases

• Grouping web analytics or log data by registrable domain instead of raw hostname
• Normalizing or validating user-submitted domains/emails before storing them
• Building cookie-scoping or same-site security checks that need the real registrable domain, not just the last two labels
• Extracting root domains from crawled URLs for deduplication in a web scraper or crawler
• Reverse-DNS-style namespacing for plugin/package identifiers derived from a domain

Under The Hood

Architecture The library separates concerns cleanly across four small modules: remote.py handles lenient netloc extraction and IPv4/IPv6 detection without raising on malformed input; cache.py implements a generic, file-locked disk cache keyed by a hash of its inputs, used both for HTTP response caching and for memoizing the parsed suffix list; suffix_list.py orchestrates fetching the PSL from a list of mirror URLs (falling back to a bundled .tld_set_snapshot file), parsing it into public/private suffix lists via regex, and threading that through the cache layer; and tldextract.py ties it together in the TLDExtract class, building a reverse-label Trie from the suffix lists and walking it label-by-label from the end of the hostname to find the longest matching suffix, correctly handling PSL wildcard (*) and exception (!label) rules. A thin cli.py argparse frontend exposes the same functionality as a command-line tool. Nothing here would break under normal use if the core Trie/suffix_index logic changed, since it’s isolated behind the TLDExtract public API.

Tech Stack Pure Python 3.10+, targeting CPython 3.10-3.14 and PyPy 3.10/3.11. Runtime dependencies are minimal and purposeful: idna for punycode decoding, requests plus requests-file for fetching suffix lists (including from local file:// URLs), and filelock for safe concurrent disk-cache access. Packaging uses setuptools with setuptools-scm for git-tag-derived versioning (no manually maintained version string). Development tooling runs through uv and tox/tox-uv for a matrix of interpreter versions.

Code Quality The test suite is extensive and multi-layered: unit tests across main_test.py, cli_test.py, custom_suffix_test.py, test_cache.py, test_parallel.py, and test_trie.py; runnable doctests embedded directly in docstrings and exercised via sybil and pytest --doctest-modules; and snapshot testing via syrupy for suffix-list parsing output. HTTP behavior is tested with the responses mocking library rather than hitting the network. mypy --strict is enforced project-wide, and ruff lint covers a broad rule set (bugbear, comprehensions, docstrings, naming, pyupgrade) in addition to formatting. CI runs the full matrix (three OSes × seven Python/PyPy versions) plus separate lint, typecheck, and codestyle jobs on every push and PR.

API Design The public surface is intentionally small — one function (extract), one class (TLDExtract), one result dataclass — but the result dataclass carries well-named derived properties (fqdn, ipv4, ipv6, reverse_domain_name, top_domain_under_public_suffix, top_domain_under_registry_suffix) that answer the follow-up questions callers usually have to write themselves. The distinction between suffix (affected by include_psl_private_domains) and registry_suffix (always the registrar-level suffix) is a genuinely useful, if subtle, piece of API design for callers who need to reason about private PSL domains like blogspot.com. The library documents, rather than silently papering over, the boundary between suffix-boundary detection and hostname validation — it explicitly does not canonicalize or validate hostnames, and the README calls out the security implications of that choice for equality checks and allowlists.