Scrapling

Scrapling is an adaptive Python web scraping framework that spans everything from a single HTTP fetch to a full concurrent, multi-session crawl. Its parser learns from a website’s structure and automatically relocates target elements when the page’s HTML changes, so scrapers built with adaptive=True keep working across redesigns instead of breaking on the next selector drift.

The framework ships multiple fetchers — a plain requests-based fetcher, a Chrome-driven fetcher, and a StealthyFetcher built to bypass anti-bot systems like Cloudflare Turnstile out of the box — plus a Spider class for scaling to concurrent crawls with pause/resume and automatic proxy rotation. It also includes a CLI, an interactive scraping shell, and an MCP server so AI coding agents can drive scraping tasks directly.

What You Get

• Multiple fetcher engines — plain HTTP (Fetcher/AsyncFetcher), full Chrome (DynamicFetcher), and an anti-bot StealthyFetcher that bypasses systems like Cloudflare Turnstile
• An adaptive parser that relocates CSS/XPath-selected elements automatically when a website’s structure changes, so scrapers survive redesigns
• A Spider crawling framework for concurrent, multi-session crawls with pause/resume and automatic proxy rotation
• A CLI and interactive scraping shell for exploring and testing selectors without writing a script first
• A built-in MCP server so AI coding agents can drive scraping and parsing tasks directly
• Real-time crawl statistics and streaming output for monitoring large-scale scraping jobs

Common Use Cases

• Scraping pages behind anti-bot protection (Cloudflare Turnstile and similar) using the StealthyFetcher without maintaining custom browser-evasion code
• Building scrapers that keep working after a target site’s HTML structure changes, using adaptive element relocation instead of brittle fixed selectors
• Running large-scale, concurrent crawls with proxy rotation and pause/resume via the Spider framework
• Giving an AI coding agent scraping capability through Scrapling’s MCP server instead of hand-writing per-site scraping tools
• Quickly prototyping and testing CSS/XPath selectors interactively via the CLI shell before scripting a full scraper

Under The Hood

Architecture: scrapling/fetchers/ (chrome.py, requests.py, stealth_chrome.py) implements the distinct fetch strategies, backed by scrapling/engines/_browsers for browser automation and engines/static.py for plain HTTP; scrapling/parser.py and scrapling/core/ (translator, mixins, storage, ai.py) implement the adaptive CSS/XPath parsing and element-relocation logic, with core/storage.py persisting the element fingerprints adaptive mode relies on. scrapling/spiders/ layers a structured crawling framework (concurrency, pause/resume, proxy rotation) on top of the fetchers, and scrapling/cli.py plus scrapling/core/shell.py provide the CLI and interactive shell entry points.

Tech Stack: Python (packaged with setuptools), integrating Playwright/Chrome-based browser automation for the dynamic and stealth fetchers alongside a plain requests-style HTTP path for lightweight fetches. The project ships a Dockerfile for containerized deployment and uses ruff for linting (ruff.toml) and tox for multi-environment testing.

Code Quality: 56 test files under tests/, GitHub Actions CI (tests badge in the README), and py.typed for typed consumption. The project maintains README translations in 9 languages and a dedicated ROADMAP.md, both signals of an actively maintained, community-oriented open-source project backed by very high GitHub activity (70k+ stars, daily commits).

API Design: The one-liner quickstart (StealthyFetcher.fetch(url) then .css(selector, adaptive=True)) gets a working, anti-bot-resistant scrape running immediately, and the same selection API is shared across all fetcher types, so switching from a plain fetch to a stealth or full-browser fetch is a one-line change. The Spider class mirrors familiar crawling-framework conventions (start_urls, parse() as an async generator), and the bundled CLI/shell plus an official agent-skill directory materially lower the barrier for both human and AI-agent-driven scraping workflows.