ort

ort is a Rust interface for performing hardware-accelerated inference and training on machine learning models in the ONNX format. Built as the spiritual successor to the now-inactive onnxruntime-rs, it is primarily a safe wrapper around Microsoft’s ONNX Runtime library, with additional support for alternative pure-Rust backends.

The library is designed to run models exported from PyTorch, TensorFlow, Keras, scikit-learn, or PaddlePaddle either on-device or in a datacenter, and supports a very wide range of hardware accelerators — CUDA, TensorRT, OpenVINO, DirectML, CoreML, ROCm, and many more — while remaining light enough to ship inside end-user applications.

What You Get

• A safe Rust API over ONNX Runtime 1.27 for loading and running ONNX models
• Support for dozens of execution providers/accelerators (CUDA, TensorRT, OpenVINO, DirectML, CoreML, ROCm, XNNPACK, and more)
• Optional training support in addition to inference-only workloads
• Integration with ndarray and half-precision types for tensor input/output
• Alternative pure-Rust backend support beyond the default ONNX Runtime binding
• fetch-models and load-dynamic features for flexible model/runtime loading

Common Use Cases

• Deploying PyTorch/TensorFlow/scikit-learn models exported to ONNX for fast on-device inference in Rust applications
• Running GPU-accelerated inference in datacenter services using CUDA or TensorRT execution providers
• Shipping ML inference inside lightweight desktop or embedded applications without a Python runtime
• Fine-tuning or training ONNX models directly from Rust using the training feature

Under The Hood

Architecture - The crate’s src/ is organized around distinct concerns: session/ handles model loading and execution session configuration, value/ implements ONNX tensor/value types, ep/ (execution providers) abstracts over the dozens of supported hardware backends, operator/ exposes custom-operator registration, and training/ layers on gradient-based fine-tuning support. The heavy lifting of actually calling into the C ONNX Runtime library is isolated in a separate ort-sys workspace member, keeping unsafe FFI bindings out of the main safe-Rust crate.

Tech Stack - ort links against ONNX Runtime 1.27 via ort-sys’s FFI bindings and supports optional integrations with ndarray (tensor types), half (fp16/bf16), and num-complex, all gated behind Cargo features so consumers only pull in what they use. The crate targets Rust edition 2024 with MSRV 1.88 and supports feature-gated load-dynamic/fetch-models for choosing how the native runtime is located or downloaded at build/runtime.

Code Quality - The project uses Codecov for coverage tracking, maintains a tests/ directory including a dedicated leak-check workspace member (verifying no native memory leaks across the FFI boundary — critical for a library wrapping a C library), and documents its MSRV via crates.io badges. The AGENTS.md file at the repo root indicates deliberate support for AI coding agents contributing to the project.

API Design - The API centers around a Session builder pattern (configure execution providers, then load a model, then run inputs through it), which keeps the common path relatively concise despite the large surface area of supported accelerators. The main complexity for newcomers is the sheer number of Cargo features controlling which execution providers and native runtime distribution strategy to use, requiring some upfront reading of the docs to pick the right feature set for a target platform.