pycurl

PycURL is a Python interface to libcurl, the multiprotocol file transfer library written in C. Like urllib, it fetches resources identified by a URL, but because it’s a thin C-extension wrapper over libcurl rather than a pure-Python implementation, it inherits libcurl’s speed and protocol breadth — HTTP, HTTPS, FTP, and many other protocols — along with advanced features such as connection reuse, multiplexed transfers via the multi/share interfaces, and detailed low-level transfer info.

The library exposes libcurl’s option-setting model almost directly (setopt/perform/getinfo), which makes it more verbose than requests but gives callers fine-grained control over timeouts, SSL verification, proxies, authentication, and streaming callbacks that pure-Python clients often can’t match. It has long been used where raw throughput or libcurl-specific behavior (e.g. exact TLS/proxy handling) matters more than ergonomics.

What You Get

• A Curl object exposing libcurl’s full setopt/perform/getinfo interface for configuring and executing transfers
• CurlMulti and CurlShare interfaces for concurrent transfers and shared connection/cookie/DNS state across multiple easy handles
• Support for HTTP, HTTPS, FTP, and the other protocols libcurl implements, not just HTTP(S)
• Streaming write/read/progress callbacks for handling large transfers without buffering the whole body in memory
• Fine-grained TLS, proxy, authentication, and timeout configuration mirroring libcurl’s native options
• Benchmarked-fast performance relative to pure-Python HTTP clients, since it’s a thin wrapper over the C library

Common Use Cases

• High-throughput web scraping or crawling where libcurl’s speed and connection reuse outperform pure-Python clients
• Applications needing protocols beyond HTTP(S) (e.g. FTP) through a single consistent client API
• Services requiring precise TLS/proxy/authentication configuration that maps directly to libcurl’s option set
• Concurrent transfer workloads using the CurlMulti interface to manage many simultaneous connections in one process

Under The Hood

Architecture - PycURL is implemented as a C extension module (src/module.c) with functional areas split across dedicated C files: src/easy.c, easyopt.c, easyperform.c, and easyinfo.c wrap libcurl’s “easy” single-transfer interface (setopt/perform/getinfo respectively), src/multi.c wraps the multi interface for concurrent transfers, src/share.c wraps shared state (cookies/DNS/connections) across handles, and src/mime.c and easyws.c add multipart-form and WebSocket support; oscompat.c/pythoncompat.c/stringcompat.c isolate platform and Python-version differences. Tech Stack - Written in C against the libcurl API (a system or bundled dependency, not a Python package), built via setup.py/pyproject.toml as a compiled extension; CI (ci.yml, ci-macos.yml, ci-windows.yml) builds and tests across Linux, macOS, and Windows, reflecting the cross-platform complexity of binding against a native library. Code Quality - The tests/ directory contains 91 test files covering the easy, multi, share, MIME, and WebSocket interfaces plus edge cases like proxies, timeouts, and callbacks; the project has run under active CI for over a decade with a long AUTHORS/ChangeLog history, and dual COPYING-LGPL/COPYING-MIT files formalize its licensing. API Design - The API deliberately mirrors libcurl’s own C option/constant naming (e.g. pycurl.URL, c.setopt(pycurl.URL, ...)), which means anyone familiar with libcurl’s documentation can use PycURL directly, at the cost of a more verbose, less Pythonic call style than requests.