abx-dl

abx-dl is a standalone command-line tool that downloads “everything” from a URL in a single command - HTML, JS, CSS, images, video, audio, subtitles, PDFs, screenshots, article text, git repositories, and more. It gives a simple one-shot CLI interface to the same plugin ecosystem that powers ArchiveBox, so users get the combined coverage of tools like yt-dlp, gallery-dl, wget, curl, and puppeteer without running each of them separately.

Under the hood it runs a headless Chrome browser alongside a collection of downloader/extractor plugins, each contributing its own output (a title, a screenshot, a rendered HTML snapshot, extracted article text, and so on) into a single index.jsonl manifest and per-plugin output directory. Dependencies for each plugin (binaries like wget, yt-dlp, or chrome) are resolved and installed automatically on first use unless disabled.

Beyond the CLI, abx-dl exposes framework-free Python APIs - PluginCatalog, PluginConfigResolver, service classes, typed events, and execute_hook() - so applications can embed the same download/extraction runtime directly instead of shelling out to the CLI.

What You Get

• A single CLI command (abx-dl <url>) that replaces having to run wget, yt-dlp, gallery-dl, curl, and puppeteer separately
• Automatic dependency installation - missing binaries like wget, chrome, or yt-dlp are installed on demand unless --no-install is passed
• Structured output per run: an index.jsonl manifest plus one subdirectory per plugin that produced output
• Selectable output by plugin name (--plugins=wget,title) or by output type/mimetype (--output=html,pdf,video/)
• A persistent, cross-run config system (abx-dl config) with per-plugin overrides and automatic alias resolution
• Embeddable Python APIs (PluginCatalog, PluginConfigResolver, typed events) for applications that want the same runtime without shelling out to the CLI

Common Use Cases

• One-off archiving of a web page’s HTML, screenshot, PDF, and readable text for later reference
• OSINT and research workflows that need a full capture of a page’s HTML, metadata, and media in one command
• Bulk downloading of video/audio/subtitles from a URL without separately invoking yt-dlp
• Digital preservation snapshots that combine wget, screenshot, and readability output for redundancy
• Embedding URL-to-artifact extraction inside a larger application via the exposed Python service classes

Under The Hood

Architecture abx-dl is built around an event-driven orchestrator (abx_dl/orchestrator.py) that wires a set of services onto a caller-provided abxbus EventBus and drives the whole lifecycle by emitting phase-root events rather than calling functions directly: an InstallEvent resolves each plugin’s required_binaries through abxpkg before anything runs, then a CrawlEvent fans out into CrawlSetupEvent (long-lived per-crawl hooks such as launching Chrome), CrawlStartEvent → SnapshotEvent (per-URL extraction hooks), and cleanup/completion events that terminate background processes. Services such as BinaryService, ProcessService, SnapshotService, ArchiveResultService, and TagService (under abx_dl/services/) each subscribe to a slice of this event tree and are composable independently, so an embedding application can attach only the listener suites it needs and dispatch typed events itself instead of using the CLI. Background hooks signal readiness via their first stdout line rather than a generic bus-level event, which keeps process orchestration decoupled from any single plugin’s implementation.

Tech Stack The tool is a Python 3.12+ package built with the pdm-backend build system, using rich-click and rich for the CLI and terminal UI, pydantic/pydantic-settings for typed configuration, platformdirs for cross-platform config paths, and psutil/requests for process and network handling. Its distinguishing dependencies are two sibling ArchiveBox-maintained packages: abxbus (the async event bus that drives the orchestrator) and abxpkg (the binary dependency resolver/installer, which projects resolved binaries into a managed environment directory). Actual downloading/extraction logic lives in the separately versioned abx-plugins package, which abx-dl loads by convention or via an ABX_PLUGINS_DIR override - abx-dl itself is the runtime/orchestration layer, not the place where individual downloader implementations live. A Dockerfile ships a containerized build for environments that want the CLI without a local Python/Chrome setup.

Code Quality The repository carries an extensive real-world test suite (ten test files covering the CLI, config, dependency resolution, executor, install phase, output files, packaging, permissions, plugins, and crawl validation) plus pytest-codeblocks, which executes the README’s own shell snippets as tests to keep documentation and behavior in sync. Static verification is enforced through ruff, mypy, pyright, and ty, wired into a .pre-commit-config.yaml and CI, and the project’s own contributor guide is explicit that mocks, monkeypatches, fake binaries, fake buses, and skipped/xfailed tests are disallowed in this codebase - tests are expected to exercise real subprocesses, real binaries, and real network calls or pytest-httpserver. Release automation bumps the version, waits for the matching test and Docker workflows to pass, and only then publishes through PyPI trusted publishing, which keeps published versions tied to a green CI run.

What Makes It Unique Rather than reimplementing scraping/downloading logic, abx-dl’s core idea is to be a thin, one-shot CLI front-end over the same plugin catalog that powers the much larger ArchiveBox project, so a single command can invoke the combined capabilities of tools like yt-dlp, gallery-dl, wget, and a headless browser without requiring a full ArchiveBox collection or database. Its binary-dependency model is also distinctive: instead of vendoring or assuming pre-installed tools, it treats every plugin’s required binary as a typed event that a separate resolver (abxpkg) can satisfy from the host system or install on demand, which lets the same plugin run consistently across a bare host, a container, or CI without bespoke setup scripts per environment.