proc-macro2

proc-macro2 is a wrapper around the Rust compiler’s built-in proc_macro crate. Where proc_macro types can only ever exist inside a running procedural macro, proc-macro2 mirrors the same TokenStream, TokenTree, Ident, Literal, Punct, Group, and Span API in types that work in any context: build scripts, regular binaries, ordinary libraries, and unit tests.

It is the foundation the modern Rust macro ecosystem is built on. Libraries like syn and quote target proc-macro2 rather than proc_macro so that token-based tooling can be developed, reused, and tested outside the narrow confines of an actual macro invocation. Inside a real proc macro it transparently delegates to the compiler; everywhere else it falls back to a self-contained pure-Rust implementation.

What You Get

• A complete mirror of the compiler’s proc_macro types — TokenStream, TokenTree, Group, Ident, Punct, Literal, and Span — usable outside a proc macro.
• Zero-cost delegation to the real compiler API when running inside a procedural macro, and a pure-Rust fallback lexer/parser everywhere else.
• The ability to unit-test macro helper logic in isolation, since proc-macro2 values can be constructed and inspected in ordinary #[test] functions.
• An optional span-locations feature exposing line/column source positions for better diagnostics.
• A no_std-compatible core with a single lightweight dependency (unicode-ident).

Common Use Cases

• Writing custom derive, attribute, and function-like procedural macros, typically alongside syn and quote.
• Building reusable token-processing libraries that need to run in build.rs or plain application code, not just inside a macro.
• Unit-testing the internal logic of a procedural macro without a full compiler macro-expansion harness.
• Generating Rust source code programmatically from build scripts and codegen tools.

Under The Hood

Architecture

proc-macro2 is structured as a thin public facade over two interchangeable backends selected at runtime. src/lib.rs (~51KB) defines the public TokenStream, TokenTree, Group, Ident, Punct, Literal, and Span types. src/detection.rs decides, once and cached, whether the real compiler proc_macro API is available; src/wrapper.rs (~31KB) bridges to it when so, while src/fallback.rs (~35KB) provides a complete standalone implementation for build.rs, binaries, and tests. The fallback path parses source text via a hand-written lexer in src/parse.rs (~29KB), with src/rustc_literal_escaper.rs handling literal escape semantics and src/rcvec.rs providing a reference-counted vector for cheap token-stream cloning. A build.rs probe compiles feature-detection snippets to gate unstable compiler APIs.

Tech Stack

Pure Rust, edition 2021, with a minimum supported Rust version of 1.71. The only runtime dependency is unicode-ident for identifier validation; the crate is #![no_std] at its core. Cargo features gate behavior: proc-macro (default) enables the compiler bridge, span-locations exposes line/column data, and a procmacro2_semver_exempt cfg opts into nightly-tracking APIs. Dev-dependencies (quote, flate2, tar, rayon, rustversion) drive the test and benchmark suites.

Code Quality

Maturity shows throughout: deny(unsafe_op_in_unsafe_fn), an extensive curated clippy allow-list, and a broad tests/ suite (test.rs, comments.rs, features.rs, marker.rs, test_fmt.rs, test_size.rs) plus a tests/ui trybuild directory, a fuzz/ target, and benches/. The dual-backend design is carefully partitioned so the two implementations stay behaviorally identical, and the code is authored and maintained primarily by David Tolnay with 1,288 commits over eight years.

API Design

The defining design choice is deliberate one-to-one fidelity with the standard library’s proc_macro API, so migrating code between the two is largely a matter of swapping the import path. Types implement the expected Display, FromStr, Clone, and iterator conversions, making token streams ergonomic to build and inspect. Documentation is thorough via docs.rs with runnable examples, and the two-line Cargo.toml setup gets users started immediately.