Magnitude

Magnitude is an open source inference engine built for running large language models on the hardware people already own. Instead of asking users to guess which model fits their machine, it profiles the chip, memory, and bandwidth first, estimates tokens-per-second for every model in its catalog, and ranks them by speed, accuracy, intelligence, and memory footprint before anything downloads.

Once a model is selected, Magnitude handles the rest of the lifecycle: downloading, hardware-specific tuning (including speculative decoding), and serving. Models load on demand when an agent needs them and unload automatically when idle or when memory pressure rises, so the desktop app can sit in the background without babysitting.

The project is aimed squarely at people already using AI coding agents locally. A one-click Connections flow wires Magnitude’s local inference server into harnesses like Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, so those tools can point at locally-served models instead of a hosted API. Everything — prompts, files, and the models themselves — stays on the machine, and once a model is downloaded no network connection is required to use it.

Magnitude ships as native desktop apps for macOS, Windows, and Linux, bundling a magnitude CLI, with the core inference engine written in Rust for performance and the surrounding orchestration (CLI, desktop app, daemon, client SDKs) written in TypeScript on Bun. It is licensed under Apache 2.0 with no paid tier or license-gated functionality.

What You Get

• Hardware profiling that estimates tokens-per-second per model before you download anything, based on your actual chip, memory, and bandwidth
• A ranked model catalog scored on speed, accuracy, intelligence, and memory usage so you can pick the right tradeoff for your machine
• Automatic model lifecycle management — models load on demand and unload when idle or when memory gets tight, with no manual process babysitting
• One-click Connections that wire the local inference server directly into agent harnesses like Claude Code, Codex, Cline, OpenCode, and Pi
• Hardware-tuned serving, including speculative decoding, set up automatically for the detected chip (Apple Silicon, NVIDIA, AMD, or CPU-only)
• A bundled magnitude CLI shipped inside the desktop app, plus native installers for macOS, Windows, and Linux
• Fully offline operation after a model is downloaded — no API keys, token costs, or rate limits, and no telemetry of prompts or files leaving the machine

Common Use Cases

• Running a local coding agent (Claude Code, Cline, OpenCode, Codex) against a locally-served model instead of a metered hosted API
• Evaluating which open-weight model actually runs well on a specific laptop or workstation before committing to a download
• Working fully offline or in network-restricted environments where prompts and files cannot leave the machine
• Squeezing better throughput out of consumer GPUs or Apple Silicon via automatic hardware-specific tuning and speculative decoding
• Avoiding per-token API costs and rate limits for high-volume local agent workflows

Under The Hood

Architecture Magnitude is a Bun/Turborepo monorepo split into two halves: a TypeScript client/orchestration layer and a separate Rust workspace under inference/ that implements the actual engine. The TypeScript side follows an explicit, documented layering — clients (cli, desktop, web) depend only on client-common and sdk; sdk is a portable Effect RPC client with no process-supervision code; daemon-management is the only layer allowed to own native child-process supervision and login registration; and acn is the server daemon that the desktop app bundles and hosts, implementing the RPC contracts declared once in acn-protocol with mandatory replay-safety annotations. This keeps client code from ever reaching directly into agent internals, providers, or protocol packages. The Rust workspace (icn-server, icn-engine, icn-hardware, icn-catalog, icn-speculative, icn-parity) is supervised as a native child process by the desktop app rather than by individual clients, with dedicated modules for CUDA driver detection, worker-process management, and memory supervision.

Tech Stack The orchestration layer runs on Bun with heavy use of the Effect-TS ecosystem (effect, @effect/platform, @effect-atom/atom, and a project-authored effect-query package) for typed RPC, state, and caching; the desktop GUI is Electron with a React/Vite renderer; the CLI and daemon share the same TypeScript packages. The actual model-serving engine is written in Rust (toolchain 1.91.1, built with clippy and rustfmt) as a set of focused crates covering hardware topology detection, model catalog/bundle building, speculative decoding, and cross-backend parity checking. The monorepo is orchestrated with Turborepo and versioned with Changesets, and CI spans a dozen-plus GitHub Actions workflows covering native builds, code-signing, and platform-specific auto-update acceptance testing for macOS, Windows, and Linux.

Code Quality The TypeScript codebase runs in strict mode throughout and carries an unusually large test suite for a desktop-app monorepo — several hundred .test.ts/.test.tsx files alongside the source they cover, run with Vitest under Bun. The Rust side enforces clippy and rustfmt via the pinned toolchain, and includes a dedicated icn-parity/icn-parity-probe pair of crates whose specific job is verifying that the Rust/CUDA inference implementation produces output consistent with a reference implementation — a correctness discipline most local-inference projects skip. CI is split into narrow, purpose-built workflows (release checks, native builds, Apple credential tests, Windows/hosted update acceptance) rather than one monolithic pipeline, and error handling on the TypeScript side leans on Effect’s typed error channels instead of ad hoc throws.

What Makes It Unique Most local-model runners either wrap llama.cpp behind a chat UI or focus purely on serving. Magnitude instead treats hardware fit as a first-class, pre-download decision — profiling CPU topology, memory, and bandwidth to estimate real tokens-per-second per model before anything is fetched — and pairs that with a parity-testing subsystem that validates its own inference output against a reference implementation across backends. It also treats connecting to an existing coding-agent harness (Claude Code, Codex, Cline, OpenCode, and others) as the primary product surface rather than an afterthought, auto-configuring those harnesses through a dedicated Connections flow instead of exposing only a generic OpenAI-compatible endpoint.

Self-Hosting

Licensing Model Apache 2.0 licensed — all features, including hardware profiling, the model catalog, speculative decoding, and every agent harness connection, are available with no license keys, paid tiers, or usage restrictions.

Self-Hosting Restrictions None found. There is no cloud-only feature set; the desktop app is the only distribution and it runs entirely on-device.

License Key Required No. Downloading and running Magnitude requires no account, license key, or API key of any kind.