agents

LiveKit Agents is the Python framework behind LiveKit’s realtime voice AI stack: it gives you an AgentServer/AgentSession programming model for building conversational, multi-modal agents that run as long-lived server-side participants in a LiveKit room, with job scheduling, dispatch, and process supervision handled for you. It ships with a pluggable STT/LLM/TTS/VAD abstraction layer (including a hosted LiveKit Inference option and provider fallback adapters) so a session can mix and match any combination of models, plus native turn-detection, telephony integration, and MCP tool support.

The framework is deliberately built for production voice agents rather than generic chatbots: multi-agent handoff, RPC/data exchange with clients, and a built-in test-and-judge harness for asserting on non-deterministic LLM behavior are all first-class. Provider integrations (OpenAI, Anthropic, Deepgram, Cartesia, AWS, Azure, and 70+ others) ship as separate livekit-plugins-* packages installed as optional extras, keeping the core framework provider-agnostic.

What You Get

• An AgentServer/AgentSession/Agent programming model for defining conversational, multi-modal agents with instructions, tools, and lifecycle hooks
• A worker process that handles job scheduling, dispatch, and per-session process/thread supervision so a crashing agent session doesn’t take down the whole worker
• Pluggable STT, LLM, TTS, and VAD interfaces — swap providers or use LiveKit Inference (a hosted, low-latency default) without changing agent code
• Built-in semantic turn detection, interruption handling, and telephony (SIP) integration for phone-based agents
• Native MCP (Model Context Protocol) support for wiring in external tool servers with one line of code
• A test framework with LLM-judge assertions for validating non-deterministic conversational behavior

Common Use Cases

• Building a voice assistant that can hold a natural, interruptible conversation over WebRTC or telephony
• Adding an AI agent that answers or makes phone calls through LiveKit’s SIP/telephony stack
• Building multi-agent workflows that hand off between specialized agents mid-conversation
• Building multimodal agents that combine voice with vision (camera/video input) or on-screen avatars
• Prototyping with LiveKit Inference and later swapping in direct provider keys (OpenAI, Deepgram, Cartesia, etc.) for cost or latency tuning

Under The Hood

Architecture A Worker (worker.py) orchestrates a pool of job executors (ipc/proc_pool.py, job_proc_executor.py, job_thread_executor.py) that run each agent session’s JobContext in an isolated, supervised process or thread, so a crash in one session doesn’t take the worker down — a supervision layer (ipc/supervised_proc.py, channel.py, proc_client.py) handles the process boundary. Inside a session, the voice/ package implements the conversational state machine: AgentSession and AgentActivity coordinate turn detection (voice/turn.py, endpointing.py), speech generation (voice/generation.py, speech_handle.py), and tool execution (voice/tool_executor.py) against pluggable STT/LLM/TTS/VAD backends defined in llm/, stt/, tts/, and vad.py, with fallback adapters (llm/fallback_adapter.py, realtime_fallback_adapter.py) providing provider resilience. Because every provider plugin and example implements against the JobContext/AgentSession interfaces, a change to that core contract ripples out across the entire plugin ecosystem.

Tech Stack Built for Python 3.10-3.15 with a hatchling build backend and uv-managed dependencies. The core framework layers LiveKit’s own WebRTC stack (livekit, livekit-api, livekit-protocol) with pydantic 2.x for schema validation, OpenTelemetry (API, SDK, OTLP exporter) and prometheus-client for tracing and metrics, aiohttp for async HTTP, and av/numpy for audio and video handling. Optional extras pull in a wide matrix of provider-specific SDKs (OpenAI, Anthropic, Deepgram, Cartesia, AWS, Azure, and dozens more) via separately versioned livekit-plugins-* packages, plus an mcp extra for Model Context Protocol tool support.

Code Quality The repository carries an extensive test suite (over 200 test files) run with pytest, pytest-asyncio-concurrent, and pytest-xdist for parallelized async testing, alongside ruff for linting/formatting and mypy with a long list of per-module override rules reflecting deliberate, gradual typing discipline. Multiple dedicated CI workflows separate unit tests, provider-specific integration tests (test-stt.yml, test-realtime.yml), and LLM-judge evals (evals.yml) from the general build/release pipeline. The README documents a built-in AgentSession.run()/.judge() harness specifically for asserting on non-deterministic LLM output, which is unusually rigorous test tooling for this domain.

What Makes It Unique Unlike generic LLM-orchestration frameworks, this one is purpose-built around a realtime media pipeline: a transformer-based semantic turn-detection model reduces interruptions, a supervised multiprocess job-executor architecture isolates session crashes, and LiveKit Inference offers a hosted, swappable STT/LLM/TTS layer alongside direct provider plugins. Multi-agent handoff, telephony integration, and the automated conversational test/judge framework are all first-class rather than bolted on, reflecting a framework designed from the start for production voice agents rather than text chatbots.