Firecrawl

Firecrawl is an open-source web data API that gives AI agents and developers a single endpoint to search, scrape, crawl, map, and interact with the web at scale. It converts any URL into clean Markdown, structured JSON, or screenshots — handling JS rendering, rotating proxies, bot detection, and rate limits transparently so you never have to think about them.

At its core, Firecrawl operates a multi-engine scraping pipeline that intelligently selects the best method for each URL: from simple HTTP fetch for static pages all the way to headless Chromium via Playwright or its proprietary Fire Engine for bot-protected sites. The result is 96% web coverage at P95 latencies under 3.5 seconds, benchmarked against millions of real pages.

Beyond basic scraping, Firecrawl exposes a natural-language Agent endpoint powered by its Spark model family. Give it a plain-English prompt — ‘Find the pricing plans for Notion’ — and the agent searches, navigates, and extracts structured data without you specifying a single URL. It also supports browser interaction sessions, allowing scrape-then-click workflows for dynamic applications like e-commerce search.

Firecrawl is available as a managed cloud service at firecrawl.dev with generous free tiers, and as a fully self-hostable stack via Docker Compose. The self-hosted version supports the full scrape, crawl, map, and search API surface, with optional Playwright-based JavaScript rendering and SearXNG-powered search — making it viable for organizations that need data sovereignty without giving up developer experience.

What You Get

• A unified REST API covering scrape, crawl, batch-scrape, map, search, and AI-agent endpoints under consistent authentication and response schemas
• Multi-engine scraping pipeline with automatic fallback across HTTP fetch, Playwright, Fire Engine (Chrome CDP and TLS client), Wikipedia Enterprise, and a semantic search index
• An AI Agent endpoint (powered by Spark models) that accepts natural-language prompts and returns structured data without requiring upfront URL knowledge
• Browser Interact sessions that let you scrape a live page and then issue AI-prompt or code-based click/scroll/type actions in the same browser context
• Full-site crawl with configurable depth, URL filtering, robots.txt compliance, sitemap-only mode, and async job polling via webhooks
• Structured JSON extraction using Zod or Pydantic schemas, including recursive and circular reference schemas, with LLM-backed deterministic fallback
• SDKs for JavaScript/TypeScript, Python, Go, Rust, Java, Ruby, PHP, .NET, and Elixir, plus an MCP server for direct agent integration
• Docker Compose self-hosted deployment with Redis, PostgreSQL (NUQ job queue), and Playwright service included out of the box

Common Use Cases

• AI agent web grounding — connecting LLM agents to real-time web content via the MCP server or Skill, so agents can answer questions about live web pages without hallucinating stale data
• Competitive intelligence pipelines — batch-scraping dozens of competitor pricing, feature, or news pages on a schedule and feeding structured JSON into a data warehouse or vector store
• RAG knowledge-base construction — crawling an entire documentation site, converting pages to Markdown, and indexing the result for retrieval-augmented generation
• E-commerce price monitoring — using the Interact endpoint to handle login flows and JavaScript-heavy product pages that block conventional scrapers
• Research automation — submitting a natural-language query to the Agent endpoint to gather, cross-reference, and structure information across multiple web sources without writing CSS selectors
• PDF and document extraction — scraping web-hosted PDFs, DOCX, and XLSX files through the same API surface as regular web pages, normalizing output to Markdown

Under The Hood

Architecture Firecrawl follows a layered service architecture built around a central job-dispatch harness that orchestrates multiple specialized microservices. The API layer (Express + TypeScript) receives requests and immediately enqueues work into a NUQ (New Universal Queue) backed by PostgreSQL and coordinated via Redis and RabbitMQ for event-driven job completion. Scrape jobs flow through a multi-engine selection system — an engpicker service evaluates URL characteristics and team flags to assign the optimal scraping backend — then results pass through a transformer pipeline (HTML normalization, LLM extract, PII redaction, diff, product detection, and search-index dispatch) before being returned synchronously or posted via webhook. Crawls add a crawler layer that tracks visited URLs in Redis, enforces robots.txt, and spawns child scrape jobs. The Agent feature proxies through an external Spark model service that orchestrates multi-step browsing autonomously. This separation of concerns — queue, engine selection, scraping, transformation, and agent reasoning — lets each layer scale independently and be replaced without touching the others.

Tech Stack The primary API server is TypeScript running on Node.js 22 with Express and BullMQ for queue management. The Playwright browser service is a separate TypeScript microservice shipping Chromium-only Docker images. A Go microservice handles HTML-to-Markdown conversion at scale, with configurable request body limits. Redis handles rate limiting, crawl-state tracking, and job result caching; PostgreSQL (via the NUQ subsystem) serves as the durable job queue with RabbitMQ as the event bus for job completion signals. Supabase is used for optional authentication and advanced logging in cloud deployments. OpenTelemetry tracing is wired throughout the API surface with configurable Sentry error sampling. Python, Rust, Go, Java, Ruby, PHP, .NET, and Elixir SDKs wrap the REST API, each maintained in the monorepo.

Code Quality The codebase demonstrates extensive TypeScript coverage with strict types throughout the controller, service, and scraping layers. Zod schemas validate all API request bodies at the boundary, with stricter additionalProperties: false enforcement added in recent releases. Testing relies on Vitest with comprehensive integration tests split across auth/no-auth/prod environments, plus end-to-end snapshot tests (snips) for API response shapes. Error handling is explicit — a typed error hierarchy (EngineError, SiteError, ActionError, DNSResolutionError, etc.) propagates structured error metadata rather than swallowed exceptions. ESLint and Prettier are configured, and CI runs on Blacksmith runners with separate audit-ci steps for dependency vulnerabilities. Active use of OpenTelemetry spans and structured Winston logging means operational visibility is a first-class concern.

What Makes It Unique Firecrawl’s most distinctive contribution is the engine-selection waterfall: rather than committing to a single scraping strategy, it maintains a ranked list of engines (index cache, proprietary Fire Engine with TLS fingerprint spoofing and Chrome CDP stealth modes, Playwright, and plain fetch) and waterfalls through them on failure, with the engpicker service learning per-domain preferences. The proprietary Fire Engine — not available to self-hosters but accessible via the cloud API — handles sophisticated bot-detection bypasses that open-source Playwright alone cannot. Layered on top is the Go-based HTML-to-Markdown service, which the team actively forks and extends (firecrawl/html-to-markdown) to produce cleaner token-efficient output than generic converters. The NUQ (New Universal Queue) is an in-house PostgreSQL/FoundationDB-backed job queue built to replace BullMQ for large-scale crawl coordination, giving self-hosters a dependency-minimal queue without external SaaS requirements.

Self-Hosting

Firecrawl is released under the GNU Affero General Public License v3.0 (AGPL-3.0). In plain terms, this means you can use, modify, and self-host Firecrawl freely — including for commercial internal use — but if you deploy a modified version as a network service accessible to others, you must release your modifications under the same AGPL license. For most teams building internal AI pipelines or RAG systems on top of a self-hosted Firecrawl instance, this distinction rarely matters. For SaaS companies that want to offer Firecrawl-based scraping as a product to their own customers, a commercial license arrangement with Mendable/Firecrawl may be advisable.

Running Firecrawl yourself requires Docker and Docker Compose. The stack includes the main API server, a Playwright-based browser service (Chromium-only, about 1 GB lighter than the full browser image), Redis, a PostgreSQL container for the NUQ job queue, and optionally RabbitMQ for production event coordination. Environment variables configure concurrency limits, proxy settings, optional Supabase authentication, and SearXNG for self-hosted search. For teams handling JavaScript-heavy pages, the Playwright service is mandatory and adds meaningful memory overhead — plan for at least 4 GB RAM for a minimal scraping setup, more if running high-concurrency crawls. You are responsible for monitoring uptime, scaling the worker pool, managing Redis eviction policies, and keeping dependencies patched against disclosed CVEs (the project actively runs audit-ci in CI, so allowlists are visible in the repo).

Compared to the managed firecrawl.dev cloud service, self-hosters do not get access to Fire Engine — the proprietary Chrome CDP and TLS-spoofing backend that achieves the 96% web coverage benchmarks. Self-hosted deployments fall back to Playwright and plain HTTP fetch, which covers the majority of the web but will struggle with aggressive bot-detection systems. The cloud tier also includes the Spark AI agent models, zero-config proxy rotation, a usage dashboard with per-API-key activity logs, SLAs, and managed upgrades. Teams that need fire-engine-level reliability or the AI Agent endpoint without operating their own infrastructure should consider the hosted service, which offers a free tier and pay-as-you-go credit pricing.