Pagefind

Pagefind is a fully static search library built for sites with tens of thousands of pages. It runs as a post-build step against your generated HTML, producing a sharded, compressed search index alongside a small JavaScript/WASM query engine and prebuilt web-component UI — so a full-text search on a large site can ship under 300kB of network payload with no search server to run.

Because Pagefind reads the HTML your static site generator already produced, it works with any framework or generator without plugins, and layers in zero-config multilingual support, per-section results, custom metadata filtering, and relevance weighting driven by inline HTML attributes rather than a separate config DSL.

What You Get

• A CLI (npx pagefind --site public) plus NodeJS and Python indexing APIs that share one underlying request/response protocol
• A chunked, compressed index format so browsers only ever fetch the small slice of the index needed for a given query
• Prebuilt <pagefind-modal-trigger> / <pagefind-modal> web components for a working search UI with no JavaScript configuration
• Zero-config multilingual indexing that detects language per page and builds separate stemmed indexes automatically
• Inline HTML data-attribute configuration for filtering, sorting, custom metadata, and relevance weighting

Common Use Cases

• Adding full-text search to a documentation site built with Hugo, Astro, Eleventy, Jekyll, or any other static site generator
• Searching very large content sites (tens of thousands of pages) where a bundled client-side index would otherwise be too heavy
• Building knowledge-base or faceted search experiences using Pagefind’s filtering and custom-metadata attributes
• Indexing non-HTML content (PDFs, JSON, subtitles) programmatically through the NodeJS or Python indexing APIs
• Running search across multiple related static sites/domains from a single Pagefind index

Under The Hood

Architecture The CLI entry point (main.rs) parses configuration via options.rs, which uses twelf to merge CLI flags, environment variables, and JSON/YAML/TOML config files into one PagefindInboundConfig. That hands off to lib.rs’s SearchState, which walks the site directory with wax globs and fossicks each matched file in parallel via rayon (the fossick module: parser.rs and splitting.rs extract text, headings, filter/sort attributes, and language). Extracted pages are built into per-language PagefindIndexes (the index module: index_words.rs, index_filter.rs, index_metadata.rs), chunked and compressed, then written to <site>/pagefind by the output module along with the prebuilt JS/CSS UI assets; an optional actix-web-backed serve.rs spins up a local preview server. A service/api.rs layer exposes the same fossick-to-output pipeline over a request/response protocol, which is what the NodeJS and Python wrapper packages drive instead of shelling out to the CLI — so changing the core index/fragment representation ripples into every language wrapper and the WASM runtime that decodes it in the browser.

Tech Stack A Rust workspace (pagefind, pagefind_stem) on edition 2021, built as a CLI via clap 4 with twelf for merged config, rayon for parallel fossicking, tokio for async I/O, wax for glob-based directory walking, and lol_html for streaming HTML parsing. Language handling combines pagefind_stem (30+ languages) with the optional charabia crate (CJK/Thai, gated behind the “extended” feature) and unicode-segmentation/unicode-normalization. Indexes are serialized with minicbor and compressed with flate2/async-compression; the browser-side query engine lives in a separate pagefind_web crate compiled to WASM. The wrappers/node and wrappers/python packages ship precompiled per-platform binaries (@pagefind/darwin-arm64, pagefind_bin, etc.) with thin JS/Python layers over the same service API.

Code Quality Correctness is validated almost entirely through a large end-to-end integration suite — 368 .toolproof.yml test files covering anchors, diacritics, exact-phrase matching, compound filtering, index chunking, and multilingual edge cases — run in CI (.github/workflows/test.yml) against the built CLI, WASM output, and both language wrappers, rather than through unit tests embedded in the Rust source. Errors are handled explicitly with anyhow::Result and bail! rather than panics, module boundaries map cleanly to pipeline stages (fossick, index, output, service), formatting is enforced via rustfmt.toml, and Dependabot keeps dependencies current.

API Design The common path needs a single flag — npx pagefind --site public — to produce a working search bundle with a zero-config web-component UI. Power users configure indexing, filtering, and relevance through inline HTML data attributes instead of a separate config format, and the NodeJS/Python APIs mirror the CLI 1:1 through the shared service protocol, so the mental model transfers across all three surfaces. Its standout choice is shipping the query engine itself as client-side WASM/JS chunked per index shard, letting a 10,000-page site be searched fully in-browser under roughly 300kB of network transfer with no query-time backend — a distinct approach from hosted engines in the same space.