wreq-util

wreq-util is a companion crate for the wreq HTTP client that supplies the pieces you’d otherwise have to hand-roll to make a Rust HTTP client look and behave like a real browser or mobile app. Its centerpiece is Emulation: a catalog of over 100 browser and OkHttp version profiles — Chrome, Edge, Opera, Firefox, and Safari (including iOS/iPad variants) plus OkHttp — each carrying the matching TLS (JA3/JA4), HTTP/2 fingerprint, and header presets a real client of that version would send. A Platform selector (Windows/macOS/Linux/Android/iOS) fine-tunes platform-specific headers and user-agent details, and Emulation::weighted_random() picks profiles using StatCounter browser-share data so traffic looks organically distributed rather than uniformly random.

Beyond emulation, the crate ships Tower middleware for wreq: DelayLayer/JitterDelayLayer add fixed or jittered pre-request delays (with a .when() predicate to target specific requests), and an optional emulation-compression feature wires up Brotli/Zlib/Zstd codecs for TLS 1.3 certificate compression. Everything is opt-in via Cargo features (emulation, emulation-serde, tower-delay, tokio-rt/compio-rt), keeping the dependency footprint minimal for consumers who only need part of it.

What You Get

• 100+ emulation profiles - Chrome, Edge, Opera, Firefox, Safari (desktop, iOS, iPad), and OkHttp version profiles, each with matching TLS/JA3/JA4 fingerprints, HTTP/2 settings, and headers.
• Weighted random emulation - Emulation::weighted_random() picks a profile using StatCounter browser/version market-share weights so generated traffic mirrors real-world distribution.
• Platform presets - Platform enum (Windows/macOS/Linux/Android/iOS) drives platform-specific sec-ch-ua-platform, mobile flags, and user-agent details.
• Tower delay middleware - DelayLayer and JitterDelayLayer add fixed or randomized jittered delays before requests, filterable with a .when() predicate.
• TLS certificate compression - optional Brotli/Zlib/Zstd CertificateCompressor implementations for TLS 1.3 certificate compression negotiation.

Common Use Cases

• Web scraping that avoids fingerprint-based blocking - pair Emulation::Chrome147 (or a random/weighted profile) with wreq::Client so scraped sites see a request that matches typical browser TLS/HTTP2/header fingerprints.
• Load and chaos testing with realistic pacing - use JitterDelayLayer to space out synthetic request bursts so a test harness behaves less like a bot hammering an endpoint.
• API clients that need to blend in with mobile traffic - select Profile::OkHttp5 with Platform::Android to emulate Android app network traffic.
• Reducing outbound bandwidth on TLS 1.3 handshakes - enable emulation-compression to negotiate certificate compression with servers that support it.

Under The Hood

Architecture The crate is organized into two independent leaf modules gated by Cargo features: emulate (src/emulate.rs plus src/emulate/profile/{chrome,firefox,safari,opera,okhttp}/{header,http2,tls}.rs) and tower (src/tower/delay/{future,layer,service}.rs), with a shared rand module for lightweight PRNG. The emulate module is generated almost entirely by a define_enum! macro (src/emulate/macros.rs) that maps each Profile variant to a concrete emulation() constructor per browser-version module, giving near-constant-cost addition of new browser versions without touching dispatch logic; Emulation (a TypedBuilder-derived struct) composes profile + platform + http2 + headers and converts to wreq’s own Emulation type via the IntoEmulation trait, so integration happens entirely through wreq’s public extension points. The tower module implements the standard Tower Layer/Service/Future triad, keeping delay behavior fully decoupled from emulation — the two features could ship as separate crates with no refactor. Because Emulation is a flat builder struct converted at the boundary, changes to wreq’s underlying emulation representation only touch into_emulation(), never the generated profile modules.

Tech Stack Rust 2024 edition (rust-version 1.98), built on wreq 0.16 as the sole required dependency. Optional deps are cleanly feature-gated: typed-builder for the Emulation builder, serde for emulation-serde, tower + pin-project-lite + tokio’s time feature for tower-delay, and brotli/flate2/zstd for TLS certificate compression codecs. Dev-dependencies pull in tokio (full), hyper/hyper-util (HTTP/1+2 server), and http for local test servers. Build tooling is plain Cargo with cargo fmt, cargo clippy --all-targets --all-features, and cargo doc --document-private-items --all-features all enforced in CI; releases are automated via release-plz publishing to crates.io. No async runtime is hard-required by the core library — tokio-rt/compio-rt features let consumers pick their runtime through wreq.

Code Quality Tests live under tests/ as integration binaries sharing a tests/support/ module that spins up a real hyper server and asserts exact header values (user-agent, sec-ch-ua, sec-ch-ua-platform) round-trip correctly per browser profile — solid black-box coverage for a fingerprinting library where wire bytes are what matters. CI runs cargo nextest, cargo clippy --all-targets --all-features, and cargo fmt --all -- --check, so lint and format are enforced rather than optional. The crate-level #![deny(unused)] and #![deny(unsafe_code)] attributes are strict correctness gates, and #![cfg_attr(test, deny(warnings))] turns warnings into hard failures under test. Naming is consistent (compressor/layer/service triads mirroring Tower conventions), and errors are surfaced through wreq’s own Result/Error types rather than reinvented locally.

API Design The public surface is intentionally small — three imports (Emulation, Platform, Profile) cover the entire browser-emulation feature, and a single .emulation(Emulation::Chrome147) call on wreq’s client builder is enough to get a realistic fingerprint. Emulation::weighted_random() is a distinctive touch, sourcing StatCounter browser/version market-share data (cited and dated in doc comments) to pick statistically representative profiles instead of naive uniform randomness, pairing each browser family only with platforms it actually ships on. The crate-level doc comment is the README itself via #![doc = include_str!("../README.md")], keeping cargo doc and GitHub in sync automatically, and the delay middleware follows idiomatic Tower Layer/.when() conventions any Tower user already recognizes. Boilerplate to get started is minimal, though the large Profile enum is easiest to browse via generated docs rather than the source directly.