CleverCSV

CleverCSV is a Python library, built at the Alan Turing Institute, that fixes one of the most common silent-failure points in data pipelines: guessing the wrong delimiter, quote character, or escape character for a CSV file. Where Python’s built-in csv.Sniffer frequently fails on non-standard files, CleverCSV combines a fast “normal form” pre-test with a data consistency measure — scoring candidate dialects by the row-length patterns and inferred data types they produce — to reach around 97% dialect-detection accuracy on real-world files, per the package’s published research paper.

The library is designed as a drop-in replacement: existing code that imports csv can switch to clevercsv with minimal changes and gain Sniffer-compatible detection plus convenience wrappers like read_table, read_dataframe, read_dicts, and their streaming variants. A C extension (cparser) handles the performance-critical parsing loop, wrapped by a pure-Python API so callers never touch the C layer directly.

Beyond the library, CleverCSV ships an optional command-line tool (clevercsv) for detecting a file’s dialect, viewing a CSV interactively, standardizing a messy file to RFC-4180, or generating the exact Python import code for a given file — useful for one-off data wrangling without writing a script first.

What You Get

• A Sniffer-compatible Detector class that works as a drop-in replacement for csv.Sniffer but succeeds on files the standard sniffer cannot parse
• High-level wrappers (read_table, read_dataframe, read_dicts, and streaming equivalents stream_table/stream_dicts) that detect the dialect and load the file in one call
• A C-accelerated parser (cparser) wrapped by a pure-Python interface, so detection stays fast on large files
• A clevercsv command-line tool with detect, view, standardize, code, and explore subcommands for working with CSV files interactively
• DictReader/DictWriter and write_table/write_dicts helpers for round-tripping tabular data through RFC-4180-compliant output
• Published-research dialect detection (97% accuracy in the authors’ benchmark) combining normal-form pre-tests with a pattern/type/row consistency score

Common Use Cases

• Loading CSV files of unknown or inconsistent origin (scraped data, user uploads, exports from varied tools) without manually specifying delimiter/quoting
• Replacing brittle csv.Sniffer calls in an existing data-ingestion pipeline with a more accurate, compatible detector
• Loading a messy CSV directly into a Pandas DataFrame via read_dataframe when pd.read_csv guesses the wrong format
• One-off exploration or cleanup of a CSV file from the command line via clevercsv detect/clevercsv explore, without writing a script
• Generating the exact Python import code needed to reliably re-load a specific CSV file via clevercsv code

Under The Hood

Architecture CleverCSV is organized as a layered detection pipeline rather than a single heuristic function: detect.py exposes the public Detector/Sniffer API and orchestrates two strategies, normal_form.py (a cheap structural pre-test using regex-based pattern matching over candidate delimiters/quote characters) and consistency.py (a fuller data-consistency measure combining pattern, type, and row-count scoring via detect_pattern.py, detect_type.py, and potential_dialects.py, with break_ties.py resolving ambiguous candidates). Performance-sensitive parsing is delegated to a small C extension (cparser.c/abstraction.c), accessed only through the cparser_util.py wrapper, keeping the C boundary narrow and the rest of the codebase pure Python. Higher-level convenience (wrappers.py, dict_read_write.py) sits on top of this core and depends only on the detector’s public interface, so the parsing strategy could change without touching the wrapper layer.

Tech Stack A Python 3.9-3.14 package (tested across CPython and PyPy per CI) with a compiled C extension built via setuptools (declared through pyproject.toml’s ext-modules, no separate setup.py). Core runtime dependencies are minimal — chardet for encoding detection, regex for pattern matching, and packaging — while the full extra pulls in pandas for DataFrame loading, tabview and wilderness for the CLI’s interactive explore/view commands, and faust-cchardet as a faster optional encoding detector. Distribution is handled by setuptools with dynamic versioning read from clevercsv/__version__.py.

Code Quality The test suite spans tests/test_unit, tests/test_integration, and tests/test_fuzz, with roughly 18 test modules covering the C parser, encoding, dialect consistency, type/pattern detection, and the CLI console, run via unittest discover in CI. GitHub Actions gates every push/PR behind black, isort, and ruff before running the test matrix across Ubuntu, macOS, and Windows on both the minimum (3.9) and latest (3.14) supported Python versions — the same tool versions are mirrored in a local .pre-commit-config.yaml. The package ships its own type stubs (stubs/ plus a py.typed marker) and is checked with a fairly strict mypy configuration (disallow_untyped_calls, disallow_incomplete_defs, strict_equality), though top-level functions are not required to be fully annotated.

What Makes It Unique Unlike most CSV-dialect detectors, which apply a small set of heuristics to guess the delimiter, CleverCSV’s detection is grounded in a peer-reviewed method: it treats every candidate dialect as producing a parse of the file, then scores candidates using the statistical regularity of the resulting row lengths and inferred cell types (the “data consistency measure”), falling back to this only when a cheaper structural “normal form” pre-test is inconclusive. This two-tier approach — fast pre-test first, statistically-grounded fallback second — is what lets it report meaningfully higher accuracy than csv.Sniffer specifically on messy, real-world files rather than clean synthetic ones.