papermill

Papermill turns a static Jupyter notebook into a parameterized, executable unit that can be run from the command line or the Python API. It locates a cell tagged parameters, injects a new injected-parameters cell with runtime values, executes every cell through nbclient, and writes back a fully rendered output notebook — while reading and writing notebooks from local disk, HTTP(S), S3, Azure Data Lake/Blob Storage, or Google Cloud Storage through a pluggable I/O layer.

Beyond a single run, papermill’s engine and language-translator registries (loaded via Python entry points) let it act as the execution primitive inside larger orchestrators such as Airflow or Kubeflow, or be swapped out for a custom engine like papermill-origami, without any change to the notebooks themselves. When a cell raises, papermill writes an anchored error message directly into the saved output notebook pointing at the failing cell and raises a typed PapermillExecutionError, so pipeline failures are debuggable by opening the resulting notebook.

What You Get

• A CLI (papermill in.ipynb out.ipynb -p key value) and a Python API (papermill.execute_notebook) for running the same parameterized notebook either way
• Parameter injection into a tagged parameters cell, with an injected-parameters cell that replaces itself cleanly on rerun
• A pluggable I/O layer (iorw.py) reading/writing notebooks from local files, HTTP(S), S3, Azure Data Lake/Blob Storage, and Google Cloud Storage
• Kernel-language translators (Python, R, Scala, Julia, and more) that correctly serialize parameter values for the notebook’s actual kernel
• An entry-point-based engine registry so third-party packages can plug in a custom execution engine without patching papermill
• In-notebook error markers plus a raised PapermillExecutionError, so a failed pipeline run is debuggable by opening the output notebook at the failing cell

Common Use Cases

• Scheduled reporting - Data teams parameterize a single analysis notebook with a date range and run it nightly via cron or Airflow to produce a fresh output notebook per day.
• ML experiment sweeps - Researchers loop over hyperparameter combinations, calling execute_notebook with a different parameters dict per run to generate one executed notebook per configuration.
• Notebook-as-pipeline-step - Orchestration tools like Kubeflow or Airflow use papermill as the execution primitive for one step in a larger DAG, treating a notebook like a parameterized function.
• Ad hoc reproducible analysis - Analysts share a notebook with a parameters cell so colleagues can rerun the same analysis against a new file, date, or account without touching the underlying logic.

Under The Hood

Architecture Papermill is organized as a thin, layered pipeline: cli.py (a Click command) and execute.py (execute_notebook, the Python-API entry point) both funnel into the same core — loading the notebook through iorw.py, injecting parameters via parameterize.py using per-language serialization from translators.py, then handing execution off to engines.py’s PapermillEngines registry, which dispatches to a named engine (the default wraps nbclient through clientwrap.py’s PapermillNotebookClient). Errors are caught, written back into the notebook with anchor cells, and re-raised as PapermillExecutionError; the final notebook is always written out through the same iorw abstraction it was read through. The engine, translator, and I/O-backend registries are each independently extensible via Python entry points, so what would break if a core abstraction changed is any third-party engine plugin (e.g. papermill-origami) registered against PapermillEngines, not the CLI or API surface itself.

Tech Stack Papermill targets Python 3.10+ and builds directly on Jupyter’s own execution stack — nbformat for reading/writing the notebook document model and nbclient for actually driving kernel execution — with click for the CLI, pyyaml for parameter files, requests for HTTP(S) I/O, tenacity for retry logic, tqdm for the progress bar, and entrypoints powering the plugin registries. Optional extras add boto3 (S3), azure-storage-blob/azure-identity/azure-datalake-store (Azure), gcsfs (GCS), pyarrow (HDFS), and pygithub. Packaging uses setuptools with a dynamically-read version, docs are built with Sphinx and the furo theme, and CI runs on GitHub Actions with CodeQL and Codecov integration.

Code Quality The papermill/tests directory carries close to one test module per source module — execution, engines, CLI, I/O backends (S3/GCS/Azure/HDFS), translators, and parameterization all have dedicated coverage, run with pytest/pytest-cov and mocked kernel/client behavior via unittest.mock. Style is enforced with ruff (pyflakes, isort, pyupgrade, pycodestyle rules) and black, wired through a pre-commit pipeline. Errors flow through a small typed exception hierarchy (PapermillException, PapermillExecutionError, PapermillMissingParameterException) rather than bare exceptions, though the codebase has no static type-checking configuration despite using type hints sparingly in signatures.

What Makes It Unique Papermill’s core idea is treating a Jupyter notebook itself as a parameterizable, executable unit — injecting a dedicated parameters/injected-parameters cell pair rather than requiring notebook authors to restructure any code, paired with in-notebook error markers that make a failed pipeline run debuggable by opening the output notebook directly at the failing cell. Its pluggable engine, translator, and I/O registries let it serve as the execution primitive for larger systems (Kubeflow, Airflow, papermill-origami) without being tied to one specific orchestrator. It isn’t inventing new execution techniques — the actual kernel work is delegated to nbclient — but it is a widely-adopted, well-established pattern for turning notebooks into pipeline steps.