unstructured

unstructured provides open-source components for ingesting and preprocessing images and text documents — PDFs, HTML, Word docs, emails, and dozens of other formats — into clean, structured elements ready for downstream ML and LLM use. Its modular partition functions and connectors form a cohesive pipeline that normalizes wildly different document types into a consistent set of Element objects (titles, narrative text, tables, list items, and more).

The library grew out of the need to feed real-world, messy documents into retrieval-augmented-generation and NLP systems, and now handles filetype detection, OCR, layout-aware PDF parsing, chunking, and metadata extraction as a single dependency. A hosted ‘Unstructured Pipelines’ product and an MCP server (‘Unstructured Transform’) exist for production-scale or agent-driven use, but the core unstructured PyPI package remains fully open source under Apache-2.0.

What You Get

• A single partition() entry point that auto-detects file type (PDF, DOCX, HTML, email, images, and 60+ others) and returns normalized Element objects
• Format-specific partitioners under unstructured/partition/ for fine-grained control (partition_pdf, partition_html, partition_docx, partition_email, etc.)
• Built-in chunking strategies (unstructured/chunking/) for splitting partitioned output into LLM-context-sized segments
• OCR and layout-aware PDF/image parsing via optional extras (unstructured[pdf]), including table structure inference
• A documented Element/metadata model (unstructured/documents/elements.py) covering titles, narrative text, tables, and data-source metadata for consistent downstream processing

Common Use Cases

• Preprocessing heterogeneous document collections (PDFs, Word docs, HTML) into a uniform format before chunking and embedding for a RAG pipeline
• Extracting clean text and structure from scanned or image-based PDFs using OCR and layout detection
• Feeding LangChain or LlamaIndex document loaders with pre-partitioned, metadata-rich elements
• Bulk-converting mixed-format document archives (emails, spreadsheets, presentations) into structured JSON for search indexing

Under The Hood

Architecture: The library centers on unstructured/partition/auto.py, which detects file type via unstructured.file_utils.filetype.detect_filetype() and dynamically imports the matching partitioner module (partition/pdf.py, partition/docx.py, partition/html/, etc.) using importlib. Every partitioner returns a list of Element subclasses defined in unstructured/documents/elements.py (e.g. Title, NarrativeText, Table), giving a single normalized data model regardless of source format. Chunking (unstructured/chunking/), cleaning (unstructured/cleaners/), and staging/export utilities (unstructured/staging/) operate on this common Element representation, and unstructured/embed/ adds optional embedding-provider integrations.

Tech Stack: Pure Python 3.11–3.13 (per pyproject.toml), with core dependencies on beautifulsoup4/lxml/html5lib for markup parsing, spacy and langdetect for NLP/language detection, python-magic/filetype for MIME detection, and unstructured-client for the hosted API. Heavier document types (PDF OCR, DOCX, PPTX, XLSX) are gated behind optional extras (unstructured[pdf], unstructured[docx], unstructured[all-docs]) rather than being hard dependencies, keeping the base install lean. Packaging uses a modern pyproject.toml with dynamic versioning.

Code Quality: Tests live in a large, mirrored test_unstructured/ tree (partition, documents, chunking, cleaners, embed, nlp, metrics, benchmarks subpackages) plus a separate test_unstructured_ingest/ package for connector tests, indicating substantial coverage across the format-specific partitioners. typings/ holds custom type stubs, and CI is Renovate-managed (renovate.json5) for dependency updates. With 147 contributors and ~1,900 commits, the codebase shows active multi-maintainer review norms.

API Design: The single partition(filename=..., strategy=...) entry point is deliberately low-ceremony — one function call handles filetype detection and dispatch for the vast majority of use cases, with format-specific functions available for power users needing per-type options (e.g. pdf_infer_table_structure, extract_images_in_pdf). The consistent Element return type across all partitioners is the library’s main ergonomic strength, letting downstream code stay format-agnostic.