httptools

httptools wraps Node.js’s battle-tested llhttp parser (and a legacy http-parser URL parser) as a Cython extension, giving Python programs a very fast, callback-driven way to parse raw HTTP request and response bytes without writing a parser by hand. Rather than exposing a high-level request/response object model, it hands the caller a HttpRequestParser or HttpResponseParser that feeds parsed events — headers, body chunks, URL, status — to a protocol object the caller implements, keeping the library unopinionated about how the parsed data is used.

Because it sidesteps Python-level parsing overhead by delegating to compiled C, httptools is the parser underneath high-throughput ASGI servers such as uvicorn, where HTTP/1.x request parsing sits directly on the request-handling hot path. It has no external runtime dependencies beyond the bundled/vendored C sources, making it a small, focused building block for anyone implementing an HTTP server or proxy in Python.

What You Get

• HttpRequestParser and HttpResponseParser classes that feed bytes in via feed_data() and invoke protocol callbacks (on_url, on_header, on_body, on_message_complete, etc.) as they parse
• A standalone parse_url() function returning a structured URL object (schema, host, port, path, query, fragment, userinfo) without needing a full parser instance
• Upgrade-request handling via HttpParserUpgrade, so WebSocket and other Upgrade flows can be detected and handed off cleanly
• should_keep_alive() and should_upgrade() helpers for connection-management decisions mid-parse
• A typed HTTPProtocol Protocol class and bundled .pyi stubs for static type-checking of the callback interface
• Optional use of a system-installed llhttp/http-parser via build flags, instead of the vendored copies, for packagers who want to manage those C libraries themselves

Common Use Cases

• Parsing incoming HTTP/1.x requests inside a custom or embedded Python web server or ASGI server implementation (as uvicorn does)
• Building a lightweight HTTP proxy or load balancer in Python that needs to inspect or forward requests without buffering a full request object
• Parsing HTTP responses when writing a low-level HTTP client on top of raw sockets
• Detecting and handling protocol Upgrade requests (e.g. WebSocket handshakes) at the transport layer before handing the connection off
• Extracting structured components from arbitrary URL strings without pulling in a heavier URL-handling dependency

Under The Hood

Architecture The library centers on a @cython.internal HttpParser base class in httptools/parser/parser.pyx that owns a malloc’d llhttp_t C struct and a llhttp_settings_t callback table; HttpRequestParser and HttpResponseParser are thin public subclasses that call _init() with HTTP_REQUEST/HTTP_RESPONSE mode. Parsing is bridged through module-level C callback functions (cb_on_header_field, cb_on_headers_complete, etc.) that llhttp invokes directly and that translate into calls on whatever Python “protocol” object the caller passed in, so the library never buffers a full request/response itself — it just streams events outward as feed_data() is called. URL parsing is a separate, independent path in url_parser.pyx wrapping the legacy http-parser C library rather than llhttp. errors.py defines a small exception hierarchy and protocol.py exposes a typing.Protocol describing the callback surface for static checking; this abstraction would break down only if llhttp’s callback ABI changed, which the vendored-submodule approach insulates against.

Tech Stack httptools is a Cython extension (parser.pyx, url_parser.pyx) built via setuptools.Extension with a custom build_ext subclass in setup.py that vendors llhttp and http-parser as git submodules under vendor/, compiling their C sources directly (-O2) unless --use-system-llhttp/--use-system-http-parser is passed. The build backend is setuptools.build_meta (declared in pyproject.toml, requiring setuptools>=80.9.0,<=82.0.1), with Cython>=3.1.0 invoked on demand via cythonize() when .c files aren’t already present. Python 3.9+ is required; dev tooling listed in pyproject.toml’s dependency group includes pyright (strict mode) and pytest, and CI (.github/workflows/) uses uv for builds and an external release-validation action gated on httptools/_version.py changes.

Code Quality Tests live in a single tests/test_parser.py (694 lines) using Python’s built-in unittest framework (with unittest.mock for callback assertions), covering request/response parsing edge cases like header folding, chunked bodies, and Upgrade requests — there’s no pytest-specific fixture usage despite pytest being a listed dev dependency. Type safety is handled via bundled .pyi stub files (parser.pyi, url_parser.pyi) and a py.typed marker rather than inline Python type hints, since the implementation itself is Cython; pyright runs in strict mode per pyproject.toml. Error handling is explicit and typed, with a dedicated exception hierarchy (HttpParserError and subclasses) rather than generic exceptions, and malloc failures in __cinit__ raise MemoryError directly. No linter/formatter config (e.g. ruff, black) is present in the repo.

API Design The public surface is deliberately minimal and callback-oriented: two parser classes plus one standalone function (parse_url), with all parsing results delivered via optional protocol methods (on_header, on_body, etc.) that the caller implements only for the events it cares about — getattr(protocol, 'on_x', None) checks mean unimplemented callbacks are simply skipped rather than requiring stub methods. This keeps boilerplate low for simple use (a client only needs feed_data() plus the callbacks it wants) but pushes all response-object construction onto the caller, which is appropriate for a low-level building block but means httptools alone doesn’t hand back a convenient parsed-request object. The HTTPProtocol typing.Protocol formalizes this contract for IDE/type-checker support without imposing an inheritance requirement.