scrubadub

scrubadub is a Python library for removing personally identifiable information (PII) from free text before it gets logged, stored, or shared. Rather than a single regex pass, it ships a Scrubber that runs a configurable set of Detector classes over the input text, each one responsible for finding one category of Filth — names, email addresses, phone numbers, credit card numbers, dates of birth, URLs, credentials, Skype and Twitter handles, and locale-specific identifiers like US/GB/CA postal codes, US and GB national insurance/tax numbers, and GB driving licence numbers.

Detected spans are passed through PostProcessor objects that decide how the Filth gets replaced — the default swaps it for a {{EMAIL}}-style placeholder token, but built-in post-processors also support prefix/suffix wrapping or outright removal, and custom ones are easy to register. Everything is locale-aware: a Scrubber(locale='en_GB') picks up only the detectors that declare support for that locale, so US and UK-specific identifiers don’t collide.

Because detectors, filth types, and post-processors are all registered through a shared catalogue-based registry rather than hardcoded, scrubadub is meant to be extended: teams write a custom Detector subclass for an internal ID format or override the replacement behavior without forking the core package. Optional companion packages (scrubadub_address, scrubadub_spacy, scrubadub_stanford) add heavier-weight detectors — address parsing and spaCy/Stanford NLP-based name detection — that aren’t bundled by default to keep the base install lightweight.

What You Get

• A Scrubber class that orchestrates detection and replacement in one call — scrubadub.clean(text) or scrubadub.clean_documents({...}) for batches
• Built-in detectors for names, emails, phone numbers, credit cards, dates of birth, URLs, credentials, Skype/Twitter handles, and locale-specific IDs (US/GB/CA postal codes, US/GB SSNs and tax numbers, GB driving licences)
• A Filth/Detector/PostProcessor class hierarchy registered via a catalogue-based plugin system, so new detectors and replacement strategies can be added without touching core code
• Locale-aware detector loading (locale='en_GB', 'en_US', etc.) so only region-relevant detectors run for a given document
• scrubadub.list_filth() to inspect exactly what was found (type, position, detector) instead of just the redacted text
• A comparison module for scoring detector accuracy against labelled datasets

Common Use Cases

• Scrubbing PII from support tickets, chat logs, or emails before they’re stored, indexed, or sent to a third-party service
• Sanitizing free-text fields before feeding them into analytics pipelines, ML training data, or LLM prompts
• Redacting names, contact details, and financial identifiers from documents prior to sharing them externally or with auditors
• Building compliance tooling (GDPR/CCPA-style data minimization) that needs a pluggable, auditable detection pipeline rather than a black-box redaction step
• Prototyping custom PII detectors for domain-specific identifiers by subclassing Detector and registering it with the catalogue

Under The Hood

Architecture The core execution path runs through scrubadub/scrubbers.py’s Scrubber class: on construction it resolves a detector_list (defaulting to every autoloading detector registered in detectors.catalogue.detector_catalogue whose supported_locale() matches the requested locale) and a post_processor_list (similarly pulled from post_processors.catalogue.post_processor_catalogue, sorted by an index attribute). clean()/iter_filth() run each detector’s iter_filth() over the text to yield Filth spans, merge overlapping spans, then pipe the results through the ordered post-processors before substitution — a clean separation between “find PII” (detectors) and “decide what to do with it” (post-processors), with scrubadub/detectors/base.py and scrubadub/post_processors/ defining the extension points. Locale-specific detectors and filth types live in en_US/en_GB subpackages under both detectors/ and filth/, keeping region logic isolated from the core dispatch loop.

Tech Stack Pure Python (99.7% of the codebase per GitHub’s language breakdown), packaged with setuptools and declaring support back to Python 3.6. Runtime dependencies are narrow and purpose-built: phonenumbers for phone detection, python-stdnum for SSN/credit-card/tax-number validation, dateparser for date-of-birth parsing, textblob for the name detector, scikit-learn for the comparison scoring module, and catalogue (the same registry library spaCy uses) for the detector/post-processor plugin system. Heavier optional detectors (address parsing, spaCy- and Stanford-NLP-based name detection) are split into separate scrubadub_address/scrubadub_spacy/scrubadub_stanford packages to keep the base install’s dependency footprint small.

Code Quality The tests/ directory contains 42 test files covering individual detectors, filth types, and the scrubber pipeline end-to-end, run via a custom tests/run.py harness (not plain pytest) and driven through tox.ini across Python 3.6-3.9. setup.cfg configures both flake8 (max line length 120) and mypy (ignore_missing_imports), and the codebase uses type hints extensively in method signatures (Optional, Sequence, Union, ClassVar throughout detectors/base.py and scrubbers.py). Docstrings follow a consistent Sphinx/reStructuredText style with runnable >>> doctests embedded in the main API functions. CI is nominally Azure Pipelines but the checked-in azure-pipelines.yml is still the unmodified starter template printing “Hello, world!” — actual test execution isn’t wired into CI as shipped, which is a real gap despite the local test suite’s breadth.

What Makes It Unique Rather than a monolithic redaction function, scrubadub treats PII detection as a plugin system: Detector, Filth, and PostProcessor are all first-class extension points registered through a shared catalogue-based registry, so adding a new identifier type or changing replacement behavior is a subclass-and-register operation rather than a fork. Combined with explicit per-locale detector gating (supported_locale()), this makes it comparatively easy to reason about exactly which detectors ran on a given document and to extend the pipeline for identifiers the maintainers never anticipated (e.g. an internal employee ID format), which is less straightforward with regex-list-style PII scrubbers.