ArchiveBox

ArchiveBox is an open-source, self-hosted web archiving solution that lets individuals, journalists, researchers, and legal professionals preserve web content in durable, non-proprietary formats—HTML, PDF, PNG, WARC, MP4, TXT, and SQLite—guaranteed to remain readable for decades without requiring ArchiveBox itself to run.

The system accepts URLs from almost any source: typed one-at-a-time, bulk-imported from bookmarks or browser history exports, pulled from RSS feeds, synced from Pocket or Pinboard, or submitted through a Chrome browser extension or REST API. Once ingested, each URL is processed through multiple extractors in parallel—Chromium for screenshots and full-page captures, wget for WARC archives, yt-dlp for media downloads, git for repository cloning, and readability for clean article text.

ArchiveBox is built around a Django backend with a CLI, self-hosted web UI, Python API, REST API (v1), and MCP server interface, making it suitable for automation pipelines, integration with other tools, and direct interactive use. All data is stored in plain files and SQLite, meaning the archive remains fully accessible through a file browser, command line, or any SQLite client without running ArchiveBox.

The project is actively developed under the MIT license, with v0.9.x in active release candidate phase introducing a pluggy-based plugin marketplace, huey task queue, Daphne ASGI server, and supervisord process management for production-grade self-hosting.

What You Get

• Multi-format snapshot capture - Archives every URL as original HTML+CSS+JS, singlefile HTML, PNG screenshot, PDF, WARC file, clean article text, page title, favicon, and HTTP headers in parallel.
• Video and audio downloading - Extracts MP3/MP4 files, subtitles, thumbnails, and metadata from YouTube, SoundCloud, and hundreds of other platforms via yt-dlp integration.
• Git repository cloning - Archives GitHub and GitLab links by cloning the full source code tree, README, and images rather than just the rendered page.
• Scheduled and automated imports - Imports from RSS feeds, Pocket, Pinboard, browser history exports, and custom sources on a cron schedule or via webhooks without manual intervention.
• Browser extension one-click archiving - The official ArchiveBox Exporter Chrome extension sends the current page directly to your self-hosted instance with a single click.
• REST API and Python API - A versioned REST API (v1) and Python library allow programmatic management of crawls, snapshots, tags, and search from external tools and scripts.
• MCP server interface - Built-in Model Context Protocol server lets AI coding assistants and LLM workflows query and submit URLs to the archive directly.
• Automatic Wayback Machine backup - By default, every archived URL is also submitted to archive.org for off-site redundancy, configurable to stealth/local-only mode.
• Login and cookie-authenticated archiving - Supports passing Chrome user profiles and cookies to archive paywalled, login-required, or session-protected pages.
• Plain-file open archive layout - All data lives in ordinary directories and SQLite; no proprietary database or running ArchiveBox instance needed to read your archive.

Common Use Cases

• Legal evidence preservation - Lawyers archive social media posts, forum threads, and web contracts as timestamped PDFs and WARC files, capturing change-detection metadata for court admissibility.
• Journalist source archiving - Reporters back up cited articles, tweets, and leaked documents before publication, ensuring sources remain verifiable if pages are deleted or paywalled.
• Academic research crawling - Researchers schedule automated daily crawls of RSS feeds and social media pages to build longitudinal datasets of text and metadata for trend analysis or LLM training.
• Personal bookmark backup with full-text search - Power users sync their Pocket or Pinboard collections to ArchiveBox so link rot never destroys their reading history, and search across the full archived text.
• Digital preservation workflows - NGOs and archivists run ArchiveBox as part of systematic web preservation pipelines, integrating with external storage like S3, NFS, or Google Drive.
• Developer API archiving - Engineers embed ArchiveBox’s REST API into content management systems, CI pipelines, and web scrapers to automatically preserve referenced URLs at publication time.

Under The Hood

Architecture ArchiveBox is structured as a Django monolith with a meaningful services layer that separates domain logic from HTTP endpoints and CLI commands. The core models—Snapshot, Crawl, ArchiveResult—progress through explicit state machines implemented with python-statemachine, making lifecycle transitions observable and auditable rather than implicit. A pluggy-based plugin system introduced in v0.9.x allows extractors to register as Python packages via hook declarations, decoupling the extraction pipeline from core without requiring framework modification. An event bus (abxbus) wires services together through typed event emissions, so the SnapshotService, CrawlService, and BinaryService communicate through published events rather than direct coupling. The architecture still relies on Django ORM and shared models as the backbone, which keeps the blast radius of cross-cutting concerns manageable but means the system is not fully decomposed into independent bounded contexts.

Tech Stack ArchiveBox runs on Python 3.13 with Django 6.0 as the web framework and Daphne as the ASGI server, supervised by supervisord for multi-process management. REST endpoints are implemented with Django-Ninja providing typed request/response schemas. Background task queuing uses huey with a SQLite broker, avoiding the need for a separate Redis or RabbitMQ dependency. Pydantic and pydantic-settings handle configuration validation and structured config models. External archiving depends on Chromium for headless browser captures, yt-dlp for media extraction, wget for WARC generation, and singlefile for standalone HTML captures—all detected and versioned at runtime via the abxpkg binary management layer. Full-text search is powered by Sonic, a lightweight Rust-based search engine. Type checking is enforced with pyright and ruff handles linting, both integrated into pre-commit hooks.

Code Quality The test suite is extensive and specifically structured, with over 140 test files covering CLI commands, REST API endpoints, admin UI interactions, migration paths, service logic, and state machine transitions. Migration tests verify schema evolution from v0.4.x through v0.9.x, catching regressions in long-running deployments. Type annotations are applied broadly across models, services, and API schemas with pyright enforcing correctness. Error handling favors explicit state transitions and service-level validation over silent catch-alls—when operations fail, state machines transition to documented failure states rather than suppressing errors. The codebase uses ruff with configured line-length and target Python version, and pre-commit hooks enforce standards on every commit. Naming is consistent across the Django model, service, admin, and CLI layers.

What Makes It Unique ArchiveBox’s most distinctive technical decision is treating every URL as a multi-extractor pipeline job rather than a single-method download, producing six to ten independent output formats per snapshot in parallel and storing all of them in plain files. The pluggy-based extractor registry allows the community to publish archiving plugins as standard PyPI packages, enabling a genuine third-party plugin ecosystem. The MCP server interface is architecturally forward-looking, positioning ArchiveBox as a first-class tool for LLM-driven research workflows where AI agents can query and submit URLs without human intermediation. UUID7-based primary keys provide time-ordered, cryptographically unique identifiers across all models, making distributed archiving and ABID-based references stable across collection merges.

Self-Hosting

ArchiveBox is released under the MIT License, which is one of the most permissive open-source licenses available. You can use it freely for personal, commercial, or organizational purposes, modify the source code without restriction, and redistribute it with or without changes. The only requirement is that the MIT copyright notice is retained in any copy or substantial portion of the software. There are no copyleft conditions, no network-use clauses, and no contributor license agreements that would affect how you deploy or integrate the software.

Running ArchiveBox yourself means you are responsible for the full operational lifecycle. The recommended deployment is Docker Compose with a persistent data volume, which handles the Python runtime, Chromium browser, yt-dlp, wget, and supporting tools in a single container. You will need to manage container updates, schedule regular backups of the SQLite database and file archive, monitor disk usage (archives grow quickly), and handle TLS termination and reverse proxying if you expose the web UI. The v0.9.x series introduces supervisord and Daphne for process management, which improves stability but means the system has multiple supervised processes to monitor. Migration from older versions can take several hours on large collections.

There is no official paid cloud tier or managed SaaS for ArchiveBox—the project does offer professional integration consulting, security review, and custom deployment support for NGOs and enterprises through its Zulip community, but these are bespoke arrangements rather than a productized service. Self-hosters rely on the GitHub issue tracker, wiki documentation, and the community Zulip for support. Compared to hosted archiving services like Conifer (Rhizome) or commercial web monitoring tools, you trade managed uptime, automatic backups, and guaranteed SLAs for complete data ownership, no per-URL fees, and unlimited retention.