GPU Hot

GPU Hot is a lightweight, web-based, self-hosted dashboard that delivers live NVIDIA GPU telemetry directly in your browser. Built for developers, MLOps teams, and data center operators who need immediate visibility into GPU utilization without routing data through cloud services or running heavyweight monitoring stacks.

The tool collects metrics via NVML (NVIDIA Management Library) with automatic per-GPU fallback to nvidia-smi for older hardware, covering utilization, temperature, memory, power draw, clock speeds, PCIe bandwidth, P-State, throttle status, and media encoder/decoder sessions. A WebSocket-based streaming model pushes updates at sub-second intervals while pausing polling entirely when no clients are connected, keeping idle CPU usage near zero.

Beyond single-machine monitoring, GPU Hot supports a hub-and-spoke deployment mode where a central hub aggregates WebSocket streams from multiple GPU nodes, enabling cluster-wide visibility from 1 to 100+ GPUs with per-device granularity preserved across all nodes. Deployment is a single Docker run command using the official GHCR image.

The v1.9.0 release adds media engine metrics (encoder/decoder utilization and sessions), a redesigned sidebar layout, modular CSS architecture, and improved error handling in the NVML collection pipeline.

What You Get

• Real-time GPU metrics - Sub-second updates for utilization, temperature, memory usage, power draw, fan speed, clock speeds (graphics/SM/memory/video), PCIe bandwidth, P-State, throttle status, and ECC error counts via NVML and nvidia-smi.
• Media engine monitoring - Encoder and decoder utilization percentages plus active session counts, enabling visibility into video transcoding and AI inference workloads that use hardware acceleration.
• Process-level tracking - Active GPU processes with PID, memory consumption, and resolved process names extracted from the command line, available when the container runs with —init —pid=host.
• Multi-GPU and multi-node aggregation - Automatic detection of all GPUs on a host and hub mode that aggregates WebSocket streams from 1 to 100+ remote nodes into a unified cluster dashboard.
• Historical time-series charts - Interactive Chart.js visualizations for utilization, temperature, power, and clock speeds with a click-to-expand correlation drawer for comparing metrics side by side.
• System-wide host metrics - CPU utilization, RAM and swap usage, load averages, network I/O rates, and disk I/O rates alongside GPU data for full host health assessment.
• REST and WebSocket APIs - GET /api/gpu-data returns a JSON snapshot; /socket.io/ streams real-time GPU, process, and system data for integration with custom dashboards, alerting tools, or CI pipelines.

Common Use Cases

• LLM inference cluster monitoring - ML engineers deploy GPU Hot in hub mode to watch GPU memory utilization and thermal throttling in real time across multiple inference servers running large language models.
• Training job optimization - Data scientists track GPU utilization over time to identify idle periods and memory pressure, adjusting batch sizes and gradient accumulation steps to maximize training throughput.
• Data center GPU resource allocation - Platform teams use the cluster dashboard to observe which nodes are underutilized before scheduling new jobs, reducing queue wait times and improving GPU density.
• CI/CD pipeline GPU health checks - DevOps engineers poll /api/gpu-data in build scripts to fail pipelines automatically if GPU memory exceeds safe thresholds or temperatures rise above defined limits before a training run.
• Video transcoding workload monitoring - Media teams use the encoder/decoder utilization metrics to verify hardware acceleration is active and identify bottlenecks in GPU-accelerated encoding pipelines.
• Pre-deployment hardware validation - System administrators validate new GPU nodes by running GPU Hot and confirming NVML metrics populate correctly before adding the node to a production cluster.

Under The Hood

Architecture GPU Hot is structured as a single FastAPI application with a clean module hierarchy that separates configuration, GPU data collection, WebSocket event handling, and multi-node hub logic into distinct files under a core/ package. Mode selection happens at startup in app.py, which conditionally imports either the monitor path (NVML + local GPU) or the hub path (WebSocket aggregation), keeping both execution paths decoupled at the handler registration level. The metrics/collector.py module organizes all NVML calls into focused methods per metric category — performance, memory, power/thermal, clocks, PCIe, media engines, and health — making it straightforward to add new metric types without touching the monitoring loop. Config is read-only at import time from environment variables, eliminating the mutable global state that plagued earlier versions. The main architectural limitation is that the monitor loop task is created inline in the WebSocket handler rather than managed as a structured lifecycle, which can make graceful shutdown under concurrent connections uneven.

Tech Stack The backend runs on Python 3.8+ with FastAPI and uvicorn, using asyncio natively throughout — NVML calls are offloaded to a thread pool via run_in_executor to avoid blocking the event loop, while hub WebSocket connections use the websockets library with full async iteration. GPU telemetry is collected via pynvml (the Python binding for NVIDIA’s NVML C library), with psutil providing host system metrics and aiohttp handling the async GitHub release check in the version API. The frontend is vanilla JavaScript with Chart.js for time-series visualization and a native WebSocket client — no framework, no bundler, no build step. CSS is modularized into design tokens, layout, and component files. Docker deployment targets nvidia/cuda:12.2.2-runtime-ubuntu22.04 and requires the NVIDIA Container Toolkit on the host. Frontend unit tests run via Vitest with jsdom; backend tests run via pytest with asyncio support.

Code Quality GPU Hot ships with a genuine dual-layer test suite: a backend pytest suite with ten unit test files covering the metrics collector, monitor, hub, handlers, config, and nvidia-smi fallback — all using unittest.mock to simulate NVML handles without hardware — and a Vitest frontend suite with seven test files covering chart configuration, WebSocket rendering, GPU card generation, and UI navigation. Error handling in the metrics collector uses specific pynvml.NVMLError catches rather than bare excepts, and each metric category degrades gracefully when a GPU does not support a given NVML query. The handlers and hub modules still use broad except Exception guards in the event loop, but these are paired with explicit error logging rather than silent swallowing. Type annotations are present on public method signatures but missing on many internal helpers. There is no static analysis toolchain (no mypy, no pylint, no flake8 configuration checked in), and no CI workflow file in the repository.

What Makes It Unique GPU Hot’s most distinctive capability is its boot-time per-GPU API detection: at startup it attempts NVML utilization collection on each GPU individually and silently switches that specific GPU to nvidia-smi mode if the metric is unavailable, without requiring any user configuration. This means a mixed fleet of modern and legacy NVIDIA cards is handled transparently in a single deployment. The media engine metrics — encoder and decoder utilization with active session counts — go significantly beyond what most open-source GPU monitors expose, reflecting the growing importance of hardware video acceleration in AI and media workloads. The hub aggregation model is architecturally unusual: rather than polling node REST endpoints on a timer, the hub maintains persistent WebSocket connections to each node and forwards the same live stream to dashboard clients, preserving sub-second latency at cluster scale without introducing a polling bottleneck. Memory change rate is computed incrementally per GPU using sample deltas, giving a leading indicator of memory leaks that raw usage charts do not provide.

Self-Hosting

GPU Hot is released under the MIT License, which is one of the most permissive open-source licenses available. You can use it commercially, modify the source, redistribute it, and embed it in proprietary products without any copyleft obligations. The only requirement is retaining the copyright notice in distributed copies. There are no open-core restrictions, no enterprise-only modules, and no license keys — the entire codebase is available under the same terms.

Running GPU Hot yourself is genuinely low-friction for the single-node case: one docker run command on any machine with the NVIDIA Container Toolkit installed, and the dashboard is live on port 1312. The hub-and-spoke cluster mode adds operational complexity — you are responsible for network reachability between hub and nodes, firewall rules on port 1312, and reconnection behavior when nodes restart. GPU Hot handles reconnection automatically with exponential backoff, but there is no built-in health alerting, authentication layer, or persistent storage for historical metrics. All telemetry lives in browser memory and is lost on page refresh. Updates require manually pulling the new container image and restarting the service.

There is no hosted or managed version of GPU Hot, no commercial support tier, and no SLA. What you gain over a fully managed GPU monitoring solution (such as Weights & Biases system monitoring, Datadog GPU integrations, or cloud provider dashboards) is zero data egress, no per-seat pricing, and direct access to low-level NVML metrics like PCIe link width, NVLink state, memory change rate, and encoder/decoder session counts that managed platforms typically do not expose. The trade-off is that reliability, retention, alerting, and multi-user access control are entirely your responsibility to build or integrate separately.