Firecrawl Python SDK

Firecrawl’s Python SDK (firecrawl-py) wraps the Firecrawl API, a hosted service that turns any URL or website into clean Markdown, HTML, or structured JSON that’s ready to feed into an LLM or AI agent. Rather than maintaining your own scraping infrastructure — headless browsers, proxy rotation, anti-bot handling, PDF/DOCX parsing — the SDK gives you typed synchronous and asynchronous clients for single-page scraping, full-site crawling, batch scraping, and web search.

Beyond generic Markdown extraction, the SDK exposes deterministic structured formats for common page types (product pages, restaurant menus, video pages) as well as LLM-based JSON extraction against a caller-supplied schema, and supports polling or a Watcher class for tracking long-running crawl and batch jobs.

What You Get

• Sync (Firecrawl) and async (AsyncFirecrawl) clients with matching method surfaces
• scrape() for single-page extraction to Markdown, HTML, or structured json/product/menu/video formats
• crawl() / start_crawl() + get_crawl_status() for full-site crawling with depth/limit controls and auto-pagination
• parse() for uploading local HTML/PDF/DOCX bytes and getting back parsed structured content
• A Watcher helper for polling long-running crawl and batch-scrape jobs without hand-rolled retry loops
• Pydantic-typed request/response models (ScrapeOptions, ParseOptions, PaginationConfig, etc.) for editor autocomplete and validation

Common Use Cases

• Feeding RAG pipelines and AI agents with clean, LLM-ready Markdown scraped from arbitrary URLs
• Crawling an entire documentation site or blog to build a searchable knowledge base
• Extracting structured product or menu data from e-commerce and restaurant pages without writing custom parsers
• Batch-scraping a list of URLs from a lead-generation or research pipeline and polling for completion
• Parsing uploaded PDFs or DOCX files into Markdown as part of a document-ingestion workflow

Under The Hood

Architecture — firecrawl-py lives in the apps/python-sdk directory of the Firecrawl monorepo (which also hosts the Node, Go, Rust, Java, .NET, Elixir, PHP, and Ruby SDKs, plus the apps/api backend service the SDK talks to). The client is versioned internally: firecrawl/v2/client.py (~1,600 lines) and firecrawl/v2/client_async.py implement the current sync/async surfaces over the v2 API, with firecrawl/v2/types.py (~1,800 lines) defining every request/response Pydantic model; a v1 compatibility path is retained alongside it. Long-running operations (crawl, batch scrape) are modeled as start-then-poll pairs (start_crawl + get_crawl_status), with a Watcher/watcher_async.py helper that wraps the poll loop.

Tech Stack — Pure Python (>=3.8) built on requests and aiohttp/httpx for sync/async HTTP, pydantic>=2.0 for request/response schemas, websockets for streaming crawl updates, and python-dotenv/nest-asyncio for environment and event-loop ergonomics. The SDK itself has no server-side dependencies — all scraping, rendering, and anti-bot logic runs on Firecrawl’s hosted API, which the SDK’s HTTP layer calls.

Code Quality — Tests live under firecrawl/__tests__/ split into unit/ and e2e/ suites (the e2e suite exercises the real v2 API), including a dedicated recursive-schema regression test for the v1 compatibility layer. The repo carries top-level AGENTS.md/CLAUDE.md files documenting conventions for AI coding agents contributing to it, and the SDK’s own LICENSE is MIT even though the parent monorepo (the self-hostable API server) is AGPL-3.0 — a split worth knowing before vendoring the server code.

API Design — The client mirrors the API’s resource model closely (scrape, crawl, batch_scrape, parse, search), keeping most calls to a single method with keyword options objects (ScrapeOptions, ParseOptions) rather than long positional signatures. Sync and async clients expose matching method names, so switching between them is a search-and-replace rather than a redesign, and typed Pydantic responses give IDE autocomplete on every field (e.g. doc.markdown, doc.video, doc.product) instead of raw dicts.