VideoHighlighter

VideoHighlighter is an open-source desktop application that finds and explains the strongest moments in raw video without uploading anything. It scores footage second by second across scene changes, motion peaks, audio levels, detected objects, recognized actions, and transcript content, then surfaces a signal timeline and a self-contained HTML report showing exactly why each moment was kept — down to the per-signal point breakdown and confidence tier.

What sets it apart from typical “AI highlights” tools is that every run explains its own arithmetic rather than handing back an opaque score. The report separates what the detectors actually observed from what the transcript merely claimed, flags moments that scored well but were still cut, and offers concrete next steps grounded in that run’s own numbers. A composition-rules engine lets users define what a combination of detections means for their footage — for example, a ball whose center falls inside a hand-labeled net region counts as a “goal” — without retraining a model or waiting on a fresh detection pass, since rules re-read detections that already exist.

The project ships two front ends over one Python engine: a PySide6 desktop GUI with a full timeline viewer, and a newer Tauri/React web-style shell backed by a FastAPI sidecar. An “Auto” pipeline can ingest an entire camera card by layout (correctly reordering GoPro’s chapter-before-file-number naming), run detection, cut a reel, and lay down a beat-synced music bed as one resumable job.

VideoHighlighter is free and AGPL-3.0 licensed for local, offline use — object detection, action recognition, audio analysis, transcript generation, CLIP-based visual search, and the composition engine are all included with no license key. A separate closed-source Pro edition adds real-time overlays, teach-by-example categories, and open-vocabulary detection as a paid upsell, positioned as an offline alternative to cloud tools like Twelve Labs, Runway, and Descript.

What You Get

• Multi-signal detection - scene changes (OpenCV), motion peaks, object detection (YOLOX via OpenVINO), action recognition, and audio peak analysis run over the same footage and feed one combined score.
• Self-explaining HTML report - every run writes a standalone report with the per-signal point breakdown for each kept clip, moments that scored well but were excluded, and a ‘said here, measured nowhere’ section that flags transcript claims no detector confirmed.
• Composition rules engine - a YAML rule format lets you define what a combination of detections (e.g. a labeled object inside a labeled region) means for your footage, scored at detector confidence and re-evaluated in milliseconds without rerunning detection.
• Local transcript and translation - OpenAI Whisper runs offline for subtitle generation, with Ollama-backed LLM translation and chapter/story summarization, so no footage or transcript ever leaves the machine.
• Card-to-film Auto pipeline - a resumable job that finds a camera card by layout, copies and verifies footage (correcting GoPro’s chapter-before-number ordering), runs detection, cuts the reel, and syncs a music bed to detected beats.
• Two front ends, one engine - a PySide6 desktop GUI with a full Timeline Viewer, and a Tauri v2 + React web shell talking to the same Python engine through a FastAPI sidecar.
• Cross-vendor GPU support - runs on Intel (OpenVINO), AMD (DirectML/ONNX Runtime), and NVIDIA (CUDA via torch) GPUs, with the object-detection stage able to fall back to a DirectML ONNX export on non-CUDA hardware.

Common Use Cases

• Sports and action footage - use composition rules (e.g. ball-in-net, ball-at-player) to score scoring or action moments beyond what a fixed 400-class action recognizer can name.
• GoPro/action-cam card dumps - run the Auto pipeline against a memory card to ingest, sort, detect, and cut a highlight reel in one resumable job, with nothing deleted from the card.
• Long-form footage triage - process hours of raw footage locally and use the signal timeline and report to jump straight to the moments worth reviewing, without uploading sensitive footage to a cloud AI service.
• Vlogs and interviews - combine transcript search with audio/motion signals to surface spoken highlights and auto-generate subtitles offline via Whisper.
• Custom-object highlight detection - label a project-specific object or region (e.g. a scoreboard or a piece of equipment) once, and drive composition rules off it instead of retraining a full classifier.

Under The Hood

Architecture main.py is a large PySide6 desktop entry point that wires up the GUI and delegates the actual work to pipeline.py, the core orchestrator that calls into action_recognition.py and object_recognition.py and coordinates a modules/ package of over 100 files handling cross-cutting concerns — audio peak analysis, motion/scene detection, caching (video_cache.py), device backend selection, ffmpeg tooling, and the composition/rule engine (compose_events.py, composition_signals.py). A separate llm/ package hosts Ollama-backed chat, translation, CLIP indexing, and clip categorization, kept apart from the deterministic detection code. A newer surface — a Tauri v2 + React frontend backed by a FastAPI sidecar (sidecar/server.py, sidecar/worker.py) — reuses the same Python engine rather than reimplementing detection, so both front ends share one pipeline as ground truth. This reads as a modular monolith rather than a services architecture: everything runs in one process (or one sidecar worker for the web shell), but individual detection kinds, export steps, and the composition engine each live in their own narrowly-scoped module. Changing the core per-second scoring representation in pipeline.py would ripple into modules/highlight_select.py, modules/auto_segments.py, and report generation, since all three consume the same score arrays.

Tech Stack Python 3.11/3.12 (both covered by the CI matrix), PySide6 for the desktop GUI, OpenCV for scene/motion detection, PyTorch for CUDA-backed action recognition alongside an OpenVINO path for Intel hardware and onnxruntime-directml for AMD, YOLOX (deliberately installed without its declared dependencies to avoid displacing the pinned torch build) for object detection, openai-whisper for local transcription, and Ollama for local LLM translation, summarization, and chat. The newer frontend is a Tauri v2 shell around a Vite/React 19 + TypeScript UI (Tailwind v4, Radix UI) that talks to a FastAPI sidecar shelling out to the same Python engine. Packaging uses PyInstaller plus Inno Setup for the Windows installer, and GitHub Actions runs the Python test suite across the version matrix alongside a separate frontend type-check/build job.

Code Quality The tests/ directory holds well over a hundred test files, several of them notably careful about isolating logic from heavy ML dependencies — one CLIP-index test suite runs entirely on synthetic embeddings so the ranking and caching math is verified in milliseconds without a GPU or the real model loaded. pytest.ini deliberately scopes collection to tests/ to keep PyInstaller build output (which imports torch at collection time) out of the suite, and CI runs it across both supported Python versions plus a frontend build check. Comments throughout the source explain non-obvious constraints rather than just restating the code — dependency-pinning rationale in requirements.txt, a documented log-line contract between the pipeline and the sidecar worker — which points to a codebase written with future contributors in mind. No Python type-checker is configured, though the TypeScript frontend runs a strict build step and its own linter; sampled modules favor explicit exception handling with logged fallbacks over silent failure.

What Makes It Unique The standout choice is pairing a self-explaining report with a composition-rules engine instead of a single opaque score. Every kept moment carries the per-signal arithmetic behind it, and a YAML rule format lets users express relations between detections that already exist — an object’s position inside a labeled region, sustained signal levels over a window — as new named events, scored at the confidence of their weakest input rather than a fixed-vocabulary classifier’s guess at something it was never trained to recognize. Because rules re-read existing detections instead of triggering new ones, tuning a rule and re-running is a near-instant loop that never invalidates the detection cache — a materially different approach from the black-box scoring typical of cloud “AI highlights” tools.

Self-Hosting

Licensing Model VideoHighlighter (this repository) is licensed under AGPL-3.0 — free to self-host and modify, with no license key or paid tier required for any feature in this codebase, including object detection, action recognition, CLIP-based visual search, the composition engine, and the training scripts.

Self-Hosting Restrictions

• None found in this repository. Everything documented in the README as part of the open-source edition (detection, transcript/translation, reel export, composition rules) is present and unrestricted in the source.

Enterprise Features

• A separate, closed-source paid edition called VideoHighlighter Pro exists alongside this open-source project, sold via a companion site (aseiel.github.io/VideoHighlighter-site). Per the README, Pro adds real-time object/action overlays during playback (including on VR side-by-side footage), teach-a-category-by-pointing with no training run, region-based “find more like this” search, open-vocabulary detection by typed keyword, and counter/scoreboard detection.
• The README states explanation itself — the scoring report, findings, and advisor — is identical in both editions and is never a paid feature.

Cloud vs Self-Hosted Not applicable — there is no hosted/cloud version; both the open-source app and Pro run locally.

License Key Required No license key is required for this AGPL-3.0 repository. The project maintains a separate Contributor License Agreement (CLA.md) that allows the same codebase’s contributions to also be offered under a private commercial license, which is how the closed-source Pro edition is legally structured, but this does not gate any feature in the open-source code.