surya

Surya is a document OCR toolkit from Datalab built around a single 650M-parameter vision-language model that jointly handles layout analysis, full-page OCR, reading order, and table recognition. Rather than chaining together separate detection, recognition, and layout models, Surya 2 routes all of these tasks through one shared VLM served locally by vllm (NVIDIA GPU) or llama.cpp (CPU/Apple Silicon), with a lightweight torch-based line-detection model kept separate for speed. It ships as a pip-installable Python package with both a CLI (surya_ocr, surya_layout, surya_table, surya_detect) and a programmatic API (RecognitionPredictor, LayoutPredictor, TableRecPredictor) for embedding OCR into applications.

The project scores competitively against much larger document-parsing models on olmOCR-bench while staying under 1B parameters, and reports an 87.2% pass rate across a 91-language internal multilingual benchmark. It’s a good fit for teams building document-processing pipelines — PDF/scanned-document ingestion, table extraction, or structured layout parsing — who want a self-hostable, open-source model rather than a hosted document-AI API, with an inference manager that transparently spawns and reuses a local model server across commands.

What You Get

• A SuryaInferenceManager that auto-detects hardware and spawns a vllm (NVIDIA GPU) or llama.cpp (CPU/Apple Silicon) backend, or attaches to an existing OpenAI-compatible server via SURYA_INFERENCE_URL
• CLI commands (surya_ocr, surya_layout, surya_table, surya_detect) that write structured JSON results keyed by input filename, covering full documents or folders of images/PDFs
• Python predictor classes (RecognitionPredictor, LayoutPredictor, TableRecPredictor, DetectionPredictor) sharing one inference manager instance for programmatic integration
• Inline math/equation recognition returned as KaTeX-compatible LaTeX inside <math> tags as part of standard OCR output, with no separate LaTeX pass required
• A --keep_server flag and SURYA_INFERENCE_KEEP_ALIVE env var to persist the spawned inference server across multiple CLI invocations, avoiding repeated model-load costs
• A bundled Streamlit app (surya_gui) for interactively trying OCR, layout, and table recognition on images or PDFs

Common Use Cases

• Extracting structured text, reading order, and layout regions from scanned PDFs or document images for downstream RAG/search pipelines
• Detecting and extracting tables (rows, columns, cells, and full HTML) from financial reports, forms, and scanned tabular documents
• Self-hosting document OCR instead of calling a hosted document-AI API, when GPU or Apple Silicon hardware is available locally
• Multilingual document digitization projects needing coverage across 91+ languages in a single model

Under The Hood

Architecture Surya’s package (surya/) is organized by task — detection/, layout/, recognition/, table_rec/, ocr_error/ — plus a shared inference/ package that owns the VLM lifecycle. inference/__init__.py defines SuryaInferenceManager, which detects hardware and instantiates one of the inference/backends/ implementations (vllm.py, llamacpp.py, openai_client.py, spawn.py) behind a shared Backend interface. Predictor classes (RecognitionPredictor, LayoutPredictor, TableRecPredictor) take the manager via explicit constructor injection rather than owning their own model instance, so one manager/server is shared across every VLM-backed task. DetectionPredictor is architecturally separate — a pure torch model with no VLM dependency — reflecting a deliberate split between the lightweight line-detection stage and the heavier VLM-backed stages. common/ holds shared utilities including a vendored detection model kept byte-identical to upstream. Because every downstream predictor depends on the same injected manager, a change to the shared Backend abstraction would ripple across layout, recognition, and table recognition simultaneously — a coupling that trades blast radius for avoiding three separate model-serving stacks.

Tech Stack A Python 3.10+ project built with hatchling and dependency-managed via uv. The core ML stack is torch/torchvision for the detection model plus transformers and huggingface-hub for model loading, with pydantic-settings powering typed, environment-overridable configuration (device auto-detection is a computed field). Inference backends are reached through an OpenAI-compatible client (openai, httpx), so the same predictor code talks to either a locally-spawned vllm server or a llama-server process without branching. Document handling goes through pypdfium2 for PDF rendering, pillow for image ops, and a version-pinned opencv-python-headless. click powers the CLI entry points declared in pyproject.toml. Dev-only dependencies include streamlit (the bundled GUI), pytest, and pre-commit; CI runs tests and PyPI publishing on every push. The deployment target is local or self-hosted inference — Docker plus the NVIDIA Container Toolkit for vllm, or a native llama-server binary for CPU/Apple Silicon — rather than a hosted API.

Code Quality The test suite covers each major subsystem (test_detection.py, test_layout.py, test_recognition.py, test_table_rec.py, test_ocr_errors.py) using a session-scoped pytest fixture that eagerly starts the VLM backend and gracefully skips VLM-dependent tests when neither vllm nor llama-server is available, rather than failing hard on GPU-less CI runners. Core files show typed function signatures and docstrings that explain non-obvious tradeoffs rather than restating the obvious, alongside defensive exception handling around subprocess and hardware-detection calls. ruff is configured for linting and formatting, with an explicit exclusion carved out for a vendored third-party module to keep it byte-identical to upstream. pre-commit is wired as a dev dependency and CI runs on every push; no dedicated static type-checker configuration was found beyond ruff.

What Makes It Unique Surya’s core bet is collapsing three traditionally separate document-AI stages — layout detection, OCR, and table structure recognition — into a single vision-language model decode, prompted differently per task, instead of chaining three specialized models as most document-AI pipelines do. It backs this with published benchmark results showing it’s competitive against models many times its parameter count, while supporting both a GPU inference backend and a CPU/Apple Silicon backend behind one interface, so the same code runs on a server or a laptop. Inline math recognition folded into standard OCR output, rather than a bolted-on separate LaTeX-OCR pass, is a similarly consolidating design choice.