pdf-inspector

pdf-inspector is a pure-Rust library that classifies PDFs as text-based, scanned, image-based, or mixed in 10-50ms, then extracts position-aware text and converts it to clean Markdown — all without OCR or machine-learning models. Built by Firecrawl to route the roughly half of real-world PDFs that already contain extractable text away from expensive OCR pipelines, it parses a document once and shares that single parse between detection and extraction, keeping full-document processing under 200ms.

Beyond the core Rust crate, the project ships first-class bindings for Python (PyO3), Node.js (napi-rs), and browser WebAssembly, plus two CLI tools — pdf2md for PDF-to-Markdown conversion and detect-pdf for classification. Its Markdown converter reconstructs headings, lists, bold/italic, code blocks, and both rectangle-based and heuristic tables, while CID/Type0 font handling and encoding-issue detection make it resilient to real-world PDFs that would otherwise produce garbled output.

What You Get

• Smart PDF classification (TextBased/Scanned/ImageBased/Mixed) in 10-50ms with a confidence score and per-page OCR routing
• Position-aware text extraction with font metadata, X/Y coordinates, and automatic multi-column reading order
• Markdown conversion covering headings, lists, bold/italic, code blocks, and dual-mode (rectangle + heuristic) table detection
• CID/Type0 font decoding via ToUnicode CMaps plus automatic broken-encoding detection so callers can fall back to OCR
• Official bindings for Python, Node.js, and browser WebAssembly, alongside pdf2md and detect-pdf CLI tools

Common Use Cases

• Pre-filtering PDFs in an ingestion pipeline so only genuinely scanned documents get sent to OCR
• Converting reports, invoices, and legal documents to clean Markdown for LLM context windows
• Extracting structured tables from financial statements and multi-page continuation tables
• Running PDF classification and extraction entirely client-side in the browser via the WASM build

Under The Hood

Architecture — pdf-inspector processes PDFs through a single shared parse: load_document_from_path/load_document_from_mem loads the lopdf::Document once, then the detector (src/detector.rs) samples content streams for Tj/TJ text operators while walking the page tree, without deserializing every object, classifying the PDF as TextBased/Scanned/ImageBased/Mixed with a confidence score. The extractor (src/extractor/) walks content streams into TextItems (content_stream.rs), resolves font/CID metadata (fonts.rs), extracts Form XObjects and hyperlinks (xobjects.rs, links.rs), and reconstructs multi-column reading order (reading_order.rs, layout.rs) via column-detection heuristics. The tables module (src/tables/) runs dual detection — detect_rects.rs for PDF drawing-operator rectangles, detect_heuristic.rs for text-alignment-based tables — into a shared grid/cell model (grid.rs) before formatting (format.rs), with financial.rs handling numeric-table edge cases and detect_struct.rs supporting structure-tagged PDFs. Finally the markdown module (src/markdown/) takes font-size statistics (analysis.rs), classifies lines as headings/lists/code/captions (classify.rs), and produces final Markdown via convert.rs and postprocess.rs. Bindings for Python (src/python.rs via PyO3), Node.js (napi/), and browser WASM (wasm/) all sit on top of the same core Rust API in lib.rs, so language-specific glue never duplicates the parsing/detection logic.

Tech Stack — Pure-Rust core (edition 2021) with a single mandatory PDF dependency, lopdf 0.41 (rayon feature for parallel parsing on native targets, wasm_js feature for browser builds without cross-origin isolation). Font handling uses ttf-parser 0.25 for Identity-H CID cmap extraction and a hand-rolled ToUnicode CMap parser (tounicode.rs) plus bundled Adobe glyph-name and Korea1 charset tables. Error handling is via thiserror 2.0, text normalization via unicode-normalization/regex/once_cell, and native builds add rayon plus env_logger/log for CLI diagnostics. Optional pyo3 0.25 (abi3-py38, extension-module) gates the python feature; the crate ships three CLI binaries (pdf2md, detect-pdf, dump_ops) with an explicit Cargo.toml include allowlist, since crates.io’s 10MiB cap otherwise forces excluding the multi-gigabyte tests/fixtures directory while still needing to ship the external/bcmaps runtime asset. Separate publish workflows push the crate to crates.io, PyPI (maturin), npm (napi-rs), and a WASM package in parallel from one source tree.

Code Quality — The repo has a 3,654-line integration test suite (tests/integration_tests.rs) covering detection, extraction, table detection, and Markdown conversion against synthetically constructed minimal PDFs rather than only fixture files, plus a Python test file (tests/test_python.py) and a Node test (napi/test.mjs) exercising each language binding independently. CI runs cargo test, a Python developer-script unittest suite, cargo fmt --check (including the separate wasm/ crate), and cargo clippy, so formatting and linting are enforced on every PR. Code documents non-obvious tradeoffs inline (e.g. a crate-level clippy allow with a one-paragraph rationale, and a comment explaining why the Cargo.toml include-list exists). Error handling uses a thiserror-derived PdfError rather than panics/unwraps in the public API, and encoding-issue detection (text_quality.rs) is a first-class concern — broken CID/Latin-1 decoding is flagged rather than silently emitting garbage text.

API Design — The public API centers on a small number of top-level functions with progressively more control — process_pdf (detect + extract + markdown in one call) at the simple end, process_pdf_with_options plus a PdfOptions builder for callers who need only classification or only extraction, and lower-level detect_pdf_type/extract_text_with_positions/to_markdown_from_items for custom pipelines. Every function has a _mem variant for in-memory byte slices alongside path-based variants, which matters for the WASM build where there’s no filesystem. Rustdoc on lib.rs includes a runnable quick-start example directly in the crate documentation, and the README duplicates equivalent quick-start snippets for Python, Node.js, browser WASM, Rust-library, and CLI usage side by side, keeping each binding’s onboarding cost low. The two CLI tools expose the same underlying logic through flags (—json, —pages, —select-pages, —compact) rather than a separate reimplementation, keeping CLI and library behavior in lockstep.