Magika

Magika is a Rust library from Google that determines file content types using a compact, highly optimized deep-learning model. It runs inference through the ONNX Runtime to classify over 200 binary and textual formats with roughly 99% accuracy, typically in about 5 milliseconds per file after the one-off model load.

Designed to be embedded directly in your own Rust code, Magika exposes a small, ergonomic API built around a reusable Session that can identify files from a path or content straight from memory, with both synchronous and asynchronous entry points.

What You Get

• A reusable Session type that loads the bundled ONNX model once and identifies many files across threads
• Synchronous and asynchronous identification from a file path or from in-memory content
• Rich results including label, description, MIME type, content group, and a confidence score for 200+ content types
• A Builder for tuning ONNX Runtime threading and graph optimization
• Optional serde support and a typed Error/Result API via thiserror

Common Use Cases

• Routing uploaded files to the right security, antivirus, or content-policy scanners
• Classifying files at scale in data pipelines where libmagic is too coarse or slow
• Detecting textual vs binary content and precise source-code languages for tooling and indexing
• Embedding accurate, near-constant-time file type detection into Rust services and CLIs

Under The Hood

Architecture The public surface in rust/lib/src/lib.rs re-exports a small set of types layered cleanly around a Session (session.rs) that wraps an ort::session::Session. A Builder (builder.rs) configures ONNX Runtime threading and graph optimization before constructing it. Identification flows through SyncInput/AsyncInput traits (input.rs) that abstract over file content, feed extracted Features into the model (model.rs), and return a FileType enum (file.rs) that distinguishes AI-inferred, rule-based, directory, and symlink results, each carrying a TypeInfo with label, description, MIME type, group, and score.

Tech Stack Written in Rust (edition 2021), it depends on ort (=2.0.0-rc.12) for ONNX Runtime inference, ndarray for tensor manipulation, tokio for async file I/O, and thiserror for error types; serde is optional behind a feature flag. The trained model ships as model.onnx bundled in the crate source, and the consuming binary is responsible for linking an ONNX Runtime.

Code Quality The crate enables missing_docs, unreachable_pub, and unused lints at warn level, and every module carries doc comments. An in-crate test module in lib.rs decodes gzip/base64 fixtures with serde-deserialized expectations, alongside a test.sh harness, giving real coverage of the identification path. Errors are modeled explicitly as an Error enum with From conversions rather than being swallowed.

API Design The API is deliberately minimal and ergonomic: Session::new() yields a ready detector, and identify_file_sync, identify_content_sync, and their async counterparts cover the common cases with one call. The builder pattern keeps advanced ONNX tuning out of the way, and doc examples in lib.rs demonstrate real usage. The main onboarding friction is the external ONNX Runtime linking requirement, which the docs address directly.