PrivateGPT

PrivateGPT is an open-source API layer that sits between your applications and locally running LLMs, providing the higher-level building blocks needed to build real AI products without cloud dependencies. Rather than running models itself, PrivateGPT connects to any OpenAI-compatible inference server and wraps it with production-grade capabilities: document ingestion and retrieval, built-in tools for web search and code execution, MCP connector support, database querying, and a skills system for reusable AI behaviors.

Version 1.0 represents a complete rebuild from its viral 2023 origins as a document chat script. The project now follows the Claude API specification as its reference model, making it a drop-in local backend for tools like Claude Code, Claude Desktop, and Microsoft 365 Office add-ins. Developers point their existing Claude API clients at PrivateGPT and get private, offline-capable inference with the same messages format, streaming, tool use, and retrieval-with-citations they expect from the hosted API.

PrivateGPT is maintained by Zylon, a company building enterprise AI infrastructure for regulated organizations. The open-source project powers Zylon’s commercial platform, which adds Kubernetes orchestration, LDAP integration, SIEM audit logs, and managed deployment on top of the same API layer.

What You Get

• Claude API-compatible messages endpoint - Exposes /v1/messages with streaming, async batch processing, token counting, and tool use following the Anthropic API specification, enabling direct substitution for Claude-backed applications without code changes.
• Document ingestion and retrieval with citations - Parses PDFs, DOCX, PPTX, XLSX, Markdown, HTML, and more using pluggable readers (MarkItDown, BeautifulSoup), chunks them into a Qdrant vector store, and returns inline citations during chat responses.
• Built-in tool suite - Ships semantic search, web search, web fetch, database querying, tabular CSV analysis via PandasAI, and sandboxed code execution as first-class tools the LLM can call during inference.
• MCP connector support - Connects to remote Model Context Protocol servers so existing MCP tool ecosystems work directly through the PrivateGPT messages API without additional proxy layers.
• Skills system - Defines reusable AI behaviors as markdown files with frontmatter, stored in object storage (local or S3), and loaded per-chat to give the LLM persistent instructions, custom tools, and persona definitions.
• Celery async processing - Offloads long-running document ingestion and batch message jobs to a Celery worker queue backed by RabbitMQ, enabling high-throughput workloads without blocking the API server.
• Pluggable inference backend - Connects to Ollama, llama.cpp, vLLM, OpenAI, Azure OpenAI, or any OpenAI-compatible endpoint via a single OPENAI_API_BASE environment variable — no code changes required to switch providers.
• Workbench UI - Provides a polished browser interface for uploading documents, sending messages, enabling tools per chat, configuring database connectors and MCP servers, and inspecting raw API requests and responses.

Common Use Cases

• Local backend for Claude Code - A developer points Claude Code at a self-hosted PrivateGPT instance to use an open-weight model for agentic terminal coding, keeping all code and context on-premise rather than sending it to Anthropic.
• Regulated-industry document assistant - A healthcare compliance team ingests internal policy documents and audit reports, then queries them through a chat interface backed by a locally running LLM, ensuring PHI never touches external servers.
• Enterprise Microsoft 365 AI add-in - An IT department deploys PrivateGPT and the Claude for Microsoft 365 add-in so employees can use AI assistance inside Word, Excel, and Outlook on private company documents using an on-premise model.
• Private database and spreadsheet analysis - A financial analyst uploads confidential CSV exports and connects PrivateGPT to an internal PostgreSQL database, then asks natural language questions that are answered by PandasAI and SQL-GPT without data leaving the VPN.
• Air-gapped environment AI platform - A defense organization runs PrivateGPT with offline mode enabled on a disconnected network, giving teams semantic search and chat capabilities over classified documents with no external network calls.
• Custom workflow automation via n8n - An operations team connects n8n workflows to PrivateGPT’s messages API to automate document processing pipelines — ingesting incoming files, extracting structured data, and routing summaries — all using local inference.

Under The Hood

Architecture PrivateGPT is organized as a clean layered application where FastAPI routers delegate to service classes, which coordinate domain components injected at startup by the Injector framework. Each capability — LLM inference, embeddings, vector storage, document ingestion, tool execution, skills, async processing — is encapsulated in its own component with an abstract interface and concrete factory-registered implementations. This means adding a new embedding provider or vector store requires implementing one class and registering it; no router or service code changes. The Celery async subsystem is a parallel track that shares the same injector and component tree, giving batch and streaming workloads access to identical business logic without duplication. The settings system uses YAML files with environment-variable interpolation and profile overlays, so the same codebase runs identically from a laptop with Ollama to a Kubernetes cluster with vLLM.

Tech Stack The server is built on Python 3.11 with FastAPI and Uvicorn, using Pydantic for all API input/output validation and Injector for dependency injection. LlamaIndex provides the document parsing, chunking, and vector retrieval core, with Qdrant as the default vector store and optional PostgreSQL integration for the node and index stores. Tool capabilities are implemented as processor classes in a pipeline: semantic search via LlamaIndex retrieval, web search through configurable provider backends, database querying via SQLAlchemy with SQLGPTParser for natural language to SQL, tabular analysis via PandasAI, and sandboxed code execution. The skills system persists markdown files to local disk or S3-compatible object storage via boto3. Celery handles async job processing with RabbitMQ as the default broker and Flower for monitoring. Observability integrates with Arize Phoenix and Opik via OpenTelemetry instrumentation.

Code Quality The project has extensive test coverage — over 100 test files covering routers, services, components, and the Celery subsystem — written with pytest and pytest-asyncio using FastAPI’s AsyncClient for integration-style HTTP tests. Tooling is comprehensive: Ruff and Black for formatting and linting with strict rules, Mypy with strict mode for type checking across the application layer, and pre-commit hooks to enforce standards before every commit. Error handling is explicit throughout, with typed exception hierarchies, Pydantic validation at API boundaries, and structured error events in the streaming protocol. The codebase is consistently typed, with py.typed declared and mypy overrides only for third-party libraries that lack stubs.

What Makes It Unique PrivateGPT’s most distinctive choice is following the Claude API specification rather than the OpenAI API as its reference model. This makes it the only open-source project that lets Claude Code, Claude Desktop, and Claude for Microsoft 365 use a local inference backend without any client modifications — a compelling niche as these tools grow. The built-in skills system — defining reusable AI behaviors as markdown files loadable per-conversation — goes meaningfully beyond most self-hosted chat APIs. The combination of native MCP connector support, built-in database querying with natural language SQL translation, and sandboxed code execution in a single deployable package is rare in the self-hosted AI space.

Self-Hosting

PrivateGPT is licensed under the Apache License 2.0, one of the most permissive open-source licenses available. You can use it commercially, modify it freely, and distribute it without any obligation to open-source your own application code. The only requirement is preserving copyright and license notices in distributed copies. There are no copyleft implications — your proprietary applications built on top of PrivateGPT remain entirely your own.

Running PrivateGPT yourself means you are responsible for the full operational stack: the inference server (Ollama, vLLM, or llama.cpp), the PrivateGPT API process, the Qdrant vector store, optionally a PostgreSQL database and RabbitMQ broker for persistence and async jobs, and the hardware to run everything. For typical deployments you need a machine with sufficient RAM and VRAM for the chosen model — commonly 16-64 GB depending on model size — plus storage for document indexes. Updates and security patches are manual, requiring you to pull new releases and redeploy. For teams without dedicated infrastructure experience, this operational burden is non-trivial.

Zylon, the commercial company behind PrivateGPT, offers an enterprise platform that adds the infrastructure layers most organizations need for production deployments: Kubernetes packaging with over 20 services, a managed inference server using NVIDIA Triton and vLLM, LDAP/Active Directory integration with RBAC, SIEM audit logs for compliance, pre-built connectors for SharePoint and Confluence, and dedicated support with SLAs. Organizations in regulated industries evaluating self-hosted PrivateGPT should weigh whether the operational and compliance requirements justify building this stack in-house versus using Zylon as a managed on-premise solution.