pymupdf4llm

PyMuPDF4LLM is a lightweight Python extension built on top of PyMuPDF that converts PDF (and, with PyMuPDF Pro, Word/Excel/PowerPoint/HWP) documents into structured output optimized for retrieval-augmented generation and LLM pipelines. A single call to to_markdown(), to_json(), or to_text() reconstructs natural reading order across single- and multi-column pages, detects and renders tables as GitHub-flavored Markdown or HTML, preserves inline formatting such as bold, italics, and code spans, and extracts or inlines embedded images — all powered by the MuPDF C engine, so there’s no GPU or cloud dependency.

What sets it apart from naive text-extraction wrappers is its hybrid OCR strategy: rather than OCR-ing every page or skipping OCR on mixed documents, it inspects each page for illegible characters, vector-graphic text simulation, existing OCR text layers, and image-embedded text, then applies OCR only to the regions that need it. This selective approach is reported to cut OCR processing time by roughly half compared to full-document OCR, while avoiding the quality degradation that comes from re-OCR-ing already-clean text.

The library ships drop-in integrations for LlamaIndex (LlamaMarkdownReader) and LangChain (via PyMuPDFLoader), supports page-level chunking with per-chunk metadata for direct ingestion into vector stores, and exposes a batch-conversion path (convert_batch) with automatic worker-count sizing based on available CPU and RAM. A pymupdf4llm CLI entry point is also installed for command-line conversion without writing any Python.

What You Get

• Three output formats - to_markdown(), to_json(), and to_text() cover LLM prompts/RAG ingestion, bbox-aware custom pipelines, and simple search indexing respectively.
• Layout-aware reading order - reconstructs natural reading order across single- and multi-column pages using a GNN-based layout model rather than naive top-to-bottom text dumps.
• Hybrid, selective OCR - inspects each page for illegible text, vector-simulated text, and embedded-text images before deciding whether to OCR, cutting OCR time roughly in half versus full-document OCR.
• Table detection with two render modes - tables are converted to GitHub-compatible Markdown by default, or to reconstructed HTML <table> elements via table_output="html".
• Page chunking for vector stores - page_chunks=True returns per-page chunk dicts with metadata, TOC items, and page boxes ready to hand to an embedding pipeline.
• Framework integrations - a LlamaMarkdownReader for LlamaIndex and a PyMuPDFLoader for LangChain ship as drop-in document loaders.
• Batch conversion with auto worker sizing - convert_batch() processes many files in parallel, sizing the worker pool from CPU count, available RAM, and OCR DPI/mode.
• CLI entry point - installs a pymupdf4llm command for converting documents without writing Python.

Common Use Cases

• RAG pipeline ingestion - a team building a retrieval-augmented chatbot converts a corpus of PDFs to chunked Markdown with page metadata, ready for an embedding model.
• Scanned document recovery - a document archive with a mix of clean digital PDFs and scanned pages runs hybrid OCR so only the image-only pages incur OCR cost.
• Structured extraction for downstream tooling - a data pipeline uses to_json() to get bounding-box and layout metadata per element instead of flat text, for custom table/field extraction.
• LlamaIndex or LangChain document loading - an app already built on LlamaIndex or LangChain drops in LlamaMarkdownReader or PyMuPDFLoader in place of a weaker default PDF loader.
• Bulk report conversion - an operations team batch-converts thousands of PDF reports overnight using convert_batch(), letting the library size the worker pool automatically.

Under The Hood

Architecture The public surface in src/__init__.py is a thin dispatcher: to_markdown(), to_json(), and to_text() each check a module-level _use_layout flag (set at import time by probing for pymupdf.layout) and route either into the modern layout pipeline (helpers/document_layout.py, via parse_document() and a returned ParsedDocument.to_markdown()/to_json()/to_text()) or a legacy helpers/pymupdf_rag.py path when the layout engine isn’t installed. table_output="html" is layered on top as an internal render_html_tables flag rather than a separate code path, keeping the layout pipeline’s text/OCR/reading-order logic intact and only swapping table rendering to helpers/table_html/. Layout inference itself is guarded by a process-wide threading.RLock (_LAYOUT_LOCK) since PyMuPDF’s layout call touches global state, and batch conversion (batch_converter.py) fans out across worker processes rather than threads, with worker_sizing.py computing a safe process count from os.cpu_count(), psutil.virtual_memory(), and OCR mode/DPI. What breaks if the core abstraction changes: the whole package hard-requires PyMuPDF’s installed version to exactly match its own VERSION_TUPLE (asserted at import time), so it is tightly version-pinned to its C-engine dependency rather than loosely coupled to it.

Tech Stack Pure Python (100% by GitHub’s language breakdown) targeting Python >=3.10, built on pymupdf and pymupdf_layout as exact-version-pinned dependencies, plus tabulate for table rendering and psutil for the worker-sizing heuristics. OCR is pluggable: it auto-detects and prefers rapidocr_onnxruntime or a Tesseract integration bundled through PyMuPDF, falling back to a warning (or an exception under force_ocr) if neither is present. The package is built with pipcl, Artifex’s own lightweight build backend (pyproject.toml declares it as the sole build requirement), rather than a mainstream tool like setuptools/hatchling/poetry, and setup.py generates a _build.py git-info file and copies src/ into the pymupdf4llm/ package at build time. A pymupdf4llm console-script entry point is registered for CLI usage.

Code Quality The tests/ directory holds 13 Python test files exercising specific numbered issues (test_137.py, test_370.py, test_sce-150.py, etc.) alongside test_ocr.py, test_table_html.py, test_tabulate.py, and a LlamaIndex-specific subdirectory — indicating regression-driven testing tied to real bug reports rather than a from-scratch unit-test suite. A test_lint.py wires up pylint (with an extensive, explicitly enumerated ignore list) and disabled flake8/codespell checks, gated behind an environment variable rather than running by default, and CI (.github/workflows/test_push.yml) runs a cross-platform matrix (Ubuntu, Windows, macOS) against an external Artifex test harness (aptest) on every PR to main rather than pytest directly in the visible repo. No obvious static type-checking (mypy/pyright) is configured. Error handling favors explicit ValueError/ImportError raises with descriptive messages (e.g. the version-mismatch check at import time) over silent fallbacks.

API Design The primary API is deliberately minimal — three top-level functions (to_markdown, to_json, to_text) that take a file path and return the target format directly, with sensible defaults (layout mode, hybrid OCR) requiring zero configuration to get useful output. Power users can reach the same knobs (force_ocr, pages, ocr_language, ocr_function, table_output, edge_threshold) as keyword arguments on the same functions rather than separate classes or builders, keeping the boilerplate to import-and-call. The one point of friction is the strict PyMuPDF version pin, which can surprise users who upgrade pymupdf independently; the README and CHANGES.md document this clearly, and error messages at import time name the exact expected vs. installed versions.