mesh-llm

Mesh LLM pools the GPUs and memory of every machine an operator owns and exposes the combined capacity as a single OpenAI-compatible API at http://localhost:9337/v1. Running mesh-llm serve --auto on a node picks a suitable backend, downloads a model if one isn’t already present, joins the best available mesh, and starts both the inference API and a web console — so a laptop, a gaming rig, and a workstation can act as one inference cluster instead of three separate, underused servers.

Two design choices set it apart from single-box local-LLM runners. Skippy stage splits let a dense model too large for any one machine’s VRAM load as contiguous transformer-layer stages spread across several peers: the coordinator plans the layer ranges, starts downstream stages first, waits for them to report ready, and only then publishes the stage-0 route, so a request for an oversized model is transparently served by a chain of machines instead of failing outright. An experimental Mixture-of-Agents gateway takes the opposite approach for a single request — sending "model": "mesh" fans the prompt out to every model currently running in the mesh, arbitrates the responses deterministically in code, and only calls out to a reducer LLM when the answers genuinely conflict.

Meshes can stay private (joined by invite token) or advertise themselves publicly through Nostr-based discovery, with an explicit owner-control plane kept separate from the public inference/gossip/routing plane so operator actions don’t need to trust the same channel as anonymous public traffic. Releases ship as native binaries for macOS, Linux (CPU, ARM64, CUDA, ROCm, Vulkan), and Windows (CPU, CUDA, ROCm, Vulkan), distributed via a companion packaging repository with Homebrew, apt, and pacman channels, checksums, SBOMs, and embedded release attestation so a packaged binary’s provenance can be verified after download.

What You Get

• A single OpenAI-compatible /v1 endpoint that transparently routes requests to whichever mesh node can serve the requested model
• A bundled web console (port 3131 by default) for inspecting mesh topology, connected nodes, and available models, or --headless to run API-only
• Skippy stage-split support for serving dense models larger than any single machine’s VRAM by sharding transformer layers across peers
• An experimental Mixture-of-Agents gateway (model: "mesh") that fans a prompt out to every model in the mesh and arbitrates the responses
• Cross-platform release binaries (macOS, Linux CPU/CUDA/ROCm/Vulkan, Windows CPU/CUDA/ROCm/Vulkan) plus Homebrew/apt/pacman packages and SDKs for Node, Kotlin, and Swift

Common Use Cases

• Pooling a home lab’s GPUs into one inference endpoint instead of running separate servers per machine
• Running local backends for coding agents (Goose, OpenCode, Claude Code, Pi) without per-token cloud billing
• Serving large open-weight models that exceed one machine’s VRAM via Skippy stage splits across several peers
• Publishing spare compute to a public, Nostr-discoverable mesh so other operators can route inference through it
• Getting ensemble-style answers from every model in a mesh at once through the Mixture-of-Agents gateway

Under The Hood

Architecture Mesh LLM is organized as an extensive Rust workspace with clear layering: a CLI crate (mesh-llm-cli) parses arguments and normalizes runtime-surface flags, a thin command-dispatch layer (commands::dispatch in the mesh-llm binary crate) handles subcommands like gpus and benchmark before the full host runtime spins up, and mesh-llm-host-runtime owns the actual orchestration (API surface, mesh membership, model management, networking, plugins, and crypto/trust policy) behind a single RuntimeOptions struct assembled from the parsed CLI. Peer routing is centralized in mesh-llm-routing’s ModelTargets/InferenceTarget types, which round-robin or sticky-route a model name to either a local port or a remote peer over iroh QUIC endpoints — every other crate that needs to reach a model goes through that one abstraction, so it is the load-bearing joint of the whole system: change its target-selection semantics and every proxy path, from the OpenAI-compatible frontend to the Skippy stage router, has to follow. Distributed model-parallel serving lives in its own skippy-* family of crates (coordinator, topology, cache, runtime, protocol), and the web console (mesh-llm-ui) and its backing mesh-llm-console-server are separate crates entirely, kept independent of the inference hot path.

Tech Stack The core is a Rust 2024-edition Cargo workspace built on Tokio (with a deliberately widened worker stack size to accommodate several large spawned futures), clap for CLI parsing, serde/serde_json for wire formats, and iroh for peer-to-peer QUIC transport — patched against a project-maintained fork for the mesh’s needs. Native model inference runs through llama.cpp-family bindings (mesh-llm-native-runtime, skippy-runtime, FFI crates for quantization), with blake3/sha2 used for hashing and release attestation. The bundled web console is a separate TypeScript stack — Vite, React, Tailwind, and Radix UI primitives, with TanStack Query/AI for data and chat state, Biome/ESLint/Prettier for linting, and Vitest plus Playwright for unit and end-to-end tests. Distribution spans native binaries across desktop and server operating systems and GPU backends, Docker images, a Fly.io deployment for the console, and companion SDKs for Node, Kotlin, and Swift.

Code Quality Test coverage is abundant and spread across nearly every crate in the workspace, backed by a dedicated test-harness crate and a separate correctness-focused crate for Skippy’s distributed serving guarantees, plus Python-based benchmark evaluations and browser end-to-end specs for the console. Continuous integration is comprehensive: a primary workflow does path-based change detection so only affected crates rebuild and test, complemented by dedicated nightly stability runs, Docker build checks, SDK smoke tests, scripted binary smoke tests, and platform-specific cache-warming jobs. Linting is enforced beyond defaults — Clippy’s cognitive-complexity and function-length thresholds are tightened repo-wide, and at least one core protocol crate forbids unsafe code outright. Error handling favors explicit Result propagation with anyhow at the entrypoint rather than silent fallbacks, and the domain is modeled with purpose-built enums (inference targets, trust policies, runtime surfaces) instead of stringly-typed state.

What Makes It Unique Two choices distinguish Mesh LLM from ordinary single-box local-model runners. Skippy stage splits let a model too large for any one machine’s memory be sharded as contiguous transformer-layer stages across multiple peers, with the coordinator sequencing startup so downstream stages are ready before the entry stage is published — turning a hard VRAM ceiling into a solvable placement problem instead of an outright failure. A separate, explicitly experimental Mixture-of-Agents gateway lets a single request fan out to every model currently in the mesh and arbitrates the responses with deterministic code rather than another model call, escalating to a reducer LLM only on genuine disagreement. Combined with peer-to-peer QUIC transport, public Nostr-based mesh discovery, and a control plane kept separate from public inference traffic, this is solving multi-machine model-parallel serving and swarm coordination — a meaningfully different problem than the single-node focus of tools like Ollama or llama.cpp.

Self-Hosting

Licensing Model Apache License 2.0 — a permissive open-source license. All functionality, including the web console, Skippy stage splitting, and the Mixture-of-Agents gateway, is available in self-hosted deployments with no license key or paid tier gating any feature.

Self-Hosting Restrictions None found. No ee/, enterprise/, pro/, or cloud/ directories exist in the repository, and no license-check or feature-flag gating was found in the source.

Enterprise Features None — there is no separate paid tier. The project is community-run infrastructure; a companion Mesh-LLM/mesh-packaging repository distributes prebuilt binaries and packages but does not gate functionality.

Cloud vs Self-Hosted Not applicable — Mesh LLM has no first-party hosted cloud offering. The mesh is peer-to-peer across machines operators run themselves; meshllm.cloud hosts documentation and the project website, not a managed inference service.

License Key Required No. No license key, activation step, or telemetry-gated feature was found anywhere in the CLI, host runtime, or web console source.