Colibri

Colibri is an inference engine, written entirely in C with zero runtime dependencies, built to run GLM-5.2 — a 744-billion-parameter mixture-of-experts model — on a machine with as little as 25GB of RAM. It does this by treating VRAM, RAM, and disk as a single managed memory hierarchy: the dense, always-needed part of the model (attention, shared experts, embeddings) stays resident in RAM at low precision, while the tens of thousands of routed experts live on disk and are streamed in on demand, based on which ones the router actually selects for a given token.

The core idea is explicitly compared to a just-in-time compiler: rather than compiling or loading an entire program up front, a JIT watches what actually executes and optimizes the hot paths. Colibri makes the analogous bet about model weights — they are not resident state to be held everywhere, but data to be staged across a heterogeneous storage hierarchy exactly when the router proves it needs them. A self-learning cache tracks which experts a given workload routes to and pins the hottest ones automatically, a lookahead thread prefetches upcoming experts by exploiting measurable structure in routing behavior, and a dual-drive mirroring scheme can pool the read bandwidth of two SSDs instead of only using the faster one.

On top of the engine sit multiple interfaces: a coli CLI with chat, one-shot run, serve, benchmarking, and diagnostic subcommands, an OpenAI-API-compatible HTTP server for dropping the model behind existing client code, and a React/TypeScript web dashboard that visualizes live token metrics, hardware tier usage, and the model’s tens of thousands of experts as an interactive 3D map of routing activity. The project backs its from-scratch reimplementation of a novel attention and MoE architecture with a rigorous validation discipline — every change must reproduce a reference implementation’s output token-for-token before merging.

Colibri spans a wide hardware range with the same codebase: on a small machine everything streams from disk at a modest but working pace, while on a large multi-GPU workstation the entire expert set can be pinned resident in VRAM, removing disk from the decode path entirely. Its explicit goal is to let a curious individual open up and modify a frontier-scale model on hardware they already own, rather than only being able to rent access to one behind an API.

What You Get

• A single-file C engine with no BLAS dependency and no Python required at runtime — just a compiled binary plus model shards.
• The coli CLI, offering interactive chat, one-shot generation, an OpenAI-compatible server mode, resource planning, installation diagnostics, quality benchmarking, and a model-conversion pipeline.
• An OpenAI-API-compatible HTTP server for wiring a self-hosted 744B model into existing client applications without changing their code.
• A React/TypeScript web dashboard with live token metrics, per-turn timing breakdowns, hardware tier usage bars, and 3D visualizations of expert routing activity.
• Prebuilt release binaries for Linux, macOS, and Windows, alongside a documented from-source build via make, Docker, and a Nix flake.
• Optional GPU acceleration through CUDA or Metal backends, chosen automatically at build time, so the same engine runs CPU-only or GPU-resident.

Common Use Cases

• Running a frontier-scale, hundreds-of-billions-of-parameter conversational model on a single personal workstation instead of a rented cloud instance.
• Serving a self-hosted large model behind an OpenAI-compatible endpoint so existing chat tooling can point at local infrastructure instead of a third-party API.
• Studying mixture-of-experts routing behavior interactively through the web dashboard’s live expert-activity visualizations.
• Scaling the same engine up to a multi-GPU workstation by pinning the full expert set resident in VRAM for higher throughput.
• Extending effective storage bandwidth on a budget by mirroring the model across two drives instead of purchasing faster storage.

Under The Hood

Architecture Colibri is a layered systems architecture wrapped around a dense, performance-critical core. A Python launcher dispatches to a compiled engine, written as a single large C file, that owns model loading, the quantized-tensor representation, mixture-of-experts routing, and the async streaming path that pulls routed experts from disk on demand. Backend selection (CPU-only, CUDA, or Metal) is abstracted behind a loader chosen at build time, so swapping the compute backend touches configuration rather than the core forward pass. An OpenAI-compatible HTTP server sits in front of the engine, and a decoupled React/TypeScript web dashboard talks only to that server’s HTTP contract, meaning the frontend can be replaced or extended without touching engine internals. The load-bearing abstraction is the expert-placement and streaming logic itself: because routing, caching, and quantization format are threaded through nearly every function in the core file, changing that abstraction would ripple through most of the engine, while everything built on top of it — server, CLI, dashboard — is comparatively insulated.

Tech Stack The engine core is pure C with threading and OpenMP for CPU parallelism, optional CUDA and Objective-C++/Metal backends for GPU acceleration, and platform-specific async I/O, deliberately built with no BLAS dependency. A Python layer wraps the engine for packaging and orchestration: a pip-installable console script, resource-planning and diagnostic helpers, and an OpenAI-compatible server module, with optional extras for model conversion, oracle-based validation against a reference implementation, and benchmarking. The web dashboard is a modern Vite, React, TypeScript, and Tailwind CSS stack with shadcn-style components, and the project additionally ships a desktop wrapper, a container image, a Nix flake, and prebuilt release binaries across the major desktop platforms.

Code Quality The project carries an extensive native test suite spanning core data structures, quantization formats, tokenization, and GPU kernels, run alongside Python-level tests through a single unified check command that continuous integration exercises across every claimed platform on every push and pull request. Beyond routine testing, the contribution process enforces a notably strict correctness bar: changes must reproduce a reference implementation’s output token-for-token before merge, which is a rare and rigorous gate for a project of this kind. Error handling in the core favors explicit, actionable failure messages and documented graceful degradation rather than silent fallbacks, and formatting and style conventions are enforced through checked-in configuration. The main tradeoff is a very dense, monolithic core file that trades easy line-by-line reviewability for the low-level control the performance goals demand.

What Makes It Unique The defining idea is treating the weights of a frontier-scale mixture-of-experts model as data to be staged across a memory hierarchy — VRAM, RAM, and disk — rather than requiring the full model to be resident, drawing an explicit analogy to how a just-in-time compiler stages only the hot paths of a program. On top of that placement model sit several genuinely novel refinements: a self-learning cache that adapts which experts stay resident based on a workload’s own history, a lookahead thread that prefetches upcoming experts by exploiting measurable structure in routing behavior, a dual-drive mirroring scheme that pools storage bandwidth across drives instead of simply replicating data, and a speculative-decoding implementation that is unusually candid about when its shortcuts do and do not pay off. The validation methodology, which forces the model’s sparse-attention mechanism to reproduce dense attention exactly and requires token-exact agreement with a reference implementation, is itself a distinctive and rigorous approach to verifying a from-scratch reimplementation of a novel architecture.

Self-Hosting

Licensing Model Apache License 2.0 — a permissive open source license. Self-hosters get full source access and may modify, redistribute, and build on the code freely, with no licensing fees, license keys, or feature restrictions of any kind.

Self-Hosting Restrictions None found. No ee/, enterprise/, pro/, or cloud/ directories exist in the repository, and no license-gating code (feature flags, license checks, “isPro”-style guards) was found anywhere in the source.

Enterprise Features None — there is no separate paid tier. Every capability documented in the repository (CLI, OpenAI-compatible server, web dashboard, GPU backends, dual-drive mirroring, model conversion) ships in the open source codebase.

Cloud vs Self-Hosted Not applicable. The project has no hosted or managed cloud offering; the GitHub Pages site is documentation only, and every capability is delivered through the self-hosted engine.

License Key Required No. Nothing in the codebase checks for or requires a license key.