arxiv.py

arxiv.py wraps the arXiv API’s Atom feed endpoint behind a small, typed Python interface. A Client handles pagination, rate-limit-friendly delays, and retry logic, while Search and Result model queries and parsed entries — including authors, affiliations, categories, links, and derived PDF/source URLs.

The library replaced its feedparser dependency with a purpose-built lxml-based Atom parser to get direct access to arXiv’s custom namespace elements (like per-author affiliations) and faster parsing, while keeping the same public Client/Search/Result surface that has powered literature-search scripts, download pipelines, and research tooling for years.

What You Get

• Client - a reusable, configurable API client that handles pagination, per-page delays, and automatic retries on HTTP errors or empty pages.
• Search - a declarative query spec supporting free-text queries, ID lists, sort criteria, and result limits, translated directly into arXiv API query parameters.
• Result - a typed representation of each paper: title, authors (with affiliations), summary, DOI, journal reference, categories, and links, plus derived pdf_url and source_url() helpers.
• Custom Atom parser - an internal lxml-based feed parser purpose-built for arXiv’s Atom/arxiv/opensearch namespaces, replacing feedparser for speed and access to nested extension elements.
• Typed exceptions - HTTPError and UnexpectedEmptyPageError, both subclasses of ArxivError, carry the failing URL and retry count for precise error handling.

Common Use Cases

• Literature search scripts - querying arXiv by keyword or author and iterating results without hand-rolling pagination or retry logic.
• Bulk paper downloads - fetching PDFs or source tarballs for a list of arXiv IDs using pdf_url and source_url().
• Research recommendation systems - fetching structured metadata (categories, abstracts, authors) as input features for downstream ranking or clustering models.
• Academic dataset construction - collecting large volumes of paper metadata into a dataset from a batch of search queries or ID lists.

Under The Hood

Architecture The library is split across two files: arxiv/__init__.py (~680 lines) holds the entire public surface — Client, Search, Result and its Author/Link inner classes, and the ArxivError hierarchy — while arxiv/_feed.py (~205 lines) isolates the internal Atom-parsing implementation behind a ParsedFeed/FeedHeader dataclass contract, importing Result lazily to avoid a circular dependency. Client.results() returns a lazy generator built from itertools.islice over Client._results(), which loops calling _parse_feed; that method retries itself recursively on HTTPError, UnexpectedEmptyPageError, or a connection error up to num_retries times, and __try_parse_feed enforces the configured request delay via a _last_request_dt check-and-sleep before each call. This keeps pagination, retry, and rate-limiting logic concentrated in Client while feed parsing stays a self-contained, swappable module.

Tech Stack Python 3.10+, with just two runtime dependencies: lxml (XML parsing, replacing a prior feedparser dependency) and requests (HTTP), plus a typing_extensions backport for pre-3.11 interpreters. The package builds with hatchling/hatch-vcs for git-tag-derived versioning and ships no server, database, or framework — it is a pure client library. Dev tooling includes ruff (with a max-complexity-10 cap), mypy in strict mode, pytest, pdoc for hosted API docs, and pip-audit for dependency vulnerability scanning.

Code Quality The test suite spans four files (test_client.py, test_result.py, test_errors.py, test_api_bugs.py) backed by a notably rigorous conftest.py: an offline-by-default fixture-cassette harness that hashes request URLs to cached JSON fixtures, supports --live/--record flags to hit the real API, and fails the suite via a pytest_sessionfinish hook if any fixture file goes unused. Error handling is explicit and typed — three custom exception classes carry URL/retry/status context, and the retry loop rethrows once retries are exhausted rather than swallowing failures. Type hints are used throughout under a strict mypy configuration, and GitHub Actions runs tests plus static analysis (ruff, mypy) on every push.

API Design The public surface is deliberately small: constructing a Client, building a Search, and calling client.results(search) covers the primary use case in three lines, demonstrated at the top of the README before any advanced configuration. Client.results() returns a generator so large result sets can be iterated without materializing them all in memory, and Search(max_results=None) opts into fetching every match. Nearly every public attribute carries an inline docstring rendered into hosted API docs, and sensible defaults (page_size=100, delay_seconds=3.0 matching arXiv’s terms of use, num_retries=3) mean most callers never need to tune the client.