SWE-bench

SWE-bench is the reference implementation of the benchmark that tests whether a language model can turn a real GitHub issue into a patch that actually fixes the bug. Instead of synthetic prompts, each task pairs a real codebase snapshot with a real issue and a hidden test suite; the model’s patch is applied and graded by running that suite inside a reproducible Docker container, so a “pass” means the code genuinely works, not that the output merely looks plausible.

The package ships as a swebench CLI (swebench infer, swebench eval, swebench report, swebench images, swebench dataset, swebench submit) built on Typer, backed by a Python library of harness, grading, and dataset-collection modules that can also be imported directly. It supports the full SWE-bench family — the original set, SWE-bench Lite, SWE-bench Verified (the 500-problem OpenAI-curated subset), SWE-bench Multimodal, and a multilingual variant — plus per-language test-log parsers for Python, JavaScript, Java, Go, PHP, Ruby, Rust, and C so results can be graded consistently across ecosystems.

Evaluation runs are containerized per-instance for reproducibility, can be built and executed locally via Docker or offloaded to Modal for cloud-scale runs, and results can be submitted straight to a Hugging Face leaderboard bucket. The project also maintains the underlying data-collection pipeline, so teams can mine their own repositories for new SWE-bench-style instances rather than relying only on the published datasets.

What You Get

• A swebench CLI with subcommands for inference (infer), evaluation (eval), report regrading (report), Docker image management (images), dataset construction/collection (dataset), and leaderboard submission (submit)
• Ready-to-run access to the full SWE-bench dataset family (full, Lite, Verified, Multimodal, Multilingual) via Hugging Face dataset aliases or local paths
• A containerized evaluation harness (swebench.harness) that applies a patch, runs the task’s test suite in Docker (or Modal for cloud runs), and classifies FAIL_TO_PASS / PASS_TO_PASS outcomes
• Per-language test-log parsers (Python, JavaScript, Java, Go, PHP, Ruby, Rust, C) so grading works consistently across the polyglot task set
• A data-collection pipeline (swebench.collect) for mining new issue/patch/test instances out of arbitrary repositories to build custom SWE-bench-style datasets
• Result submission tooling that publishes a completed run straight to a Hugging Face leaderboard bucket via swebench submit hf

Common Use Cases

• Benchmarking a new or fine-tuned LLM’s real-world coding ability against the SWE-bench Verified or Lite subsets before publishing results
• Running reproducible, containerized regression grading for an internal coding-agent evaluation pipeline instead of hand-rolling patch-apply-and-test scripts
• Mining an organization’s own repositories to build a domain-specific SWE-bench-style dataset via the dataset collect/build commands
• Re-grading previously captured evaluation logs offline with swebench report, without re-running any containers
• Submitting a completed evaluation run to a Hugging Face bucket so it can appear on public or internal SWE-bench leaderboards

Under The Hood

Architecture The package is organized as a thin Typer-based CLI layer (swebench/cli/*.py) that wires user-facing subcommands to a set of purpose-built internal packages: swebench.collect mines issues/patches/tests out of arbitrary repos, swebench.image_builder constructs the per-instance Docker images, swebench.harness (the largest module, centered on run_evaluation.py and grading.py) actually applies patches and runs tests inside containers, and swebench.submit publishes finished runs to Hugging Face. The top-level swebench/__init__.py deliberately avoids eager imports, using a __getattr__-based lazy-import table so that import swebench stays cheap and doesn’t pull in heavyweight optional dependencies (bs4, docker, datasets, modal) unless the caller actually touches a name that needs them — a change made specifically to fix slow, fragile imports (tracked as issue #525), and locked in place by a dedicated test_public_api.py regression test.

Tech Stack Built for Python 3.10+ and distributed with a uv.lock, indicating uv-managed dependency resolution. The CLI is built on Typer with Rich-formatted help panels; container orchestration goes through the docker SDK with an optional Modal backend for cloud-scale evaluation; GitPython and ghapi handle repository and GitHub API access during data collection; dataset I/O runs through Hugging Face’s datasets/huggingface_hub; patch parsing uses unidiff; and optional extras pull in transformers, torch, litellm, openai, and anthropic for the inference side, kept out of the core install so a bare evaluation setup doesn’t require a full ML stack.

Code Quality The tests/ directory covers roughly 30 files spanning CLI behavior, grading logic, per-language log-parser correctness (dedicated test files for Python, JS, Java, Go — via test_log_parsers_*.py), infra-failure classification, Docker exec streaming, and inference/litellm usage, run through pytest with coverage and a GitHub Actions pytest.yaml workflow. Source comments show close attention to grading edge cases — for example, grading.py’s SUITE_RAN regex is deliberately built (and commented) to distinguish “a test runner produced zero results because it never ran” from “it ran and everything passed,” a subtle scoring bug the authors explicitly guard against. Linting and formatting are enforced via a ruff pre-commit hook, and the code uses modern Python type hints (dict[str, Any] | None, dataclass-style typed specs) throughout.

API Design The CLI exposes a small, memorable surface (infer, eval, report, images, dataset, submit) with worked examples baked directly into --help output, and dataset arguments accept convenient aliases (full, verified, multimodal, multilingual) alongside raw Hugging Face IDs or local paths. The Python-level public API is intentionally small and explicitly enumerated in __all__, backed by a lazy-loading __getattr__ so importing the package doesn’t force in every optional dependency — a deliberate, tested trade-off between import speed and API discoverability that most benchmark-harness packages don’t bother to make.