Pipecat

Pipecat is an open-source Python framework for building real-time voice and multimodal conversational agents. It provides a pipeline-based architecture that lets developers wire together audio/video transports, speech-to-text, LLM inference, text-to-speech, and turn-detection services into a single low-latency streaming pipeline, without hand-rolling the WebRTC, buffering, and interruption-handling plumbing that real-time voice apps require.

Beyond single-agent pipelines, Pipecat supports multi-agent systems where specialist agents hand off, fan out in parallel, or coordinate over a shared bus, running locally or distributed across processes and machines. With 190+ integrations across transports (Daily, LiveKit, WebRTC, telephony), AI providers (OpenAI, Anthropic, ElevenLabs, Deepgram, and more), and a CLI for scaffolding new projects, it’s widely used as the orchestration layer for voice assistants, AI companions, and business agents that need to sound and feel like a real conversation.

What You Get

• A pipeline/frame-processor runtime for composing streaming audio, video, and text processing stages with built-in interruption and turn-taking handling
• 190+ first-party service integrations spanning transports (Daily, LiveKit, WebRTC, telephony/SIP), STT/TTS/LLM providers, and vision models
• Multi-agent primitives (parallel pipelines, service switchers, a shared event bus) for building systems where specialist agents hand off or fan out
• A pipecat CLI with pipecat init quickstart project scaffolding, agent templates, and a local dev/test runner
• Built-in metrics, observers, and evaluation hooks for tracing pipeline behavior and measuring conversation quality
• Protobuf-based frame serialization for cross-process/distributed deployment of pipeline stages

Common Use Cases

• Building a real-time voice assistant that connects a phone/WebRTC call to an LLM with low-latency, interruptible speech
• Building multi-agent customer support or intake bots where a router agent hands off to specialists
• Adding a voice or video interface to an existing LLM application without building the streaming/transport layer from scratch
• Prototyping AI companions, meeting assistants, or interactive storytelling experiences with generative audio/video
• Running structured, flow-driven dialog systems (guided intake forms, IVR replacements) with deterministic conversation logic

Under The Hood

Architecture: Pipecat’s core abstraction is the FrameProcessor, chained into a Pipeline (in src/pipecat/pipeline/) that pushes typed Frame objects (src/pipecat/frames/frames.py, also mirrored as protobuf messages for distributed use) downstream and upstream. Transports, STT/TTS/LLM services (src/pipecat/services/, one subpackage per provider — anthropic, assemblyai, aws, azure, etc.) and processors (aggregators, filters, audio processors under src/pipecat/processors/) all implement the same frame-processor interface, so pipelines are built by composing a list of these objects in pipeline.py. parallel_pipeline.py and service_switcher.py implement the multi-agent fan-out/handoff primitives, while runner.py and task.py drive execution and lifecycle. Turn-taking lives in src/pipecat/turns/, and cross-process distribution is handled by the protobuf frame format in frames/protobufs/.

Tech Stack: Pure Python (requires-python >= 3.11), built on asyncio with aiohttp for networking, pydantic for typed configuration and frame schemas, numpy/resampy/soxr/pyloudnorm for audio DSP, onnxruntime for the bundled local smart-turn (VAD/turn) analyzer, and protobuf for the wire format. Packaging uses setuptools with setuptools_scm for version derivation, and dependencies for each optional service integration are exposed as extras rather than bundled by default.

Code Quality: The repo ships 191 test files under tests/, GitHub Actions CI (badge-verified in the README) and Codecov coverage tracking. The codebase is organized into clearly bounded subpackages by responsibility (transports, services, processors, pipeline, frames), uses py.typed for typed-package distribution, and includes AGENTS.md/CLAUDE.md files documenting conventions for AI-assisted contributions — a signal of active maintainer attention to code consistency.

API Design: The pipeline-of-processors model gives a consistent mental model across very different service types (an STT service and a TTS service both just process frames), which keeps the public API small once you understand frames and processors. The pipecat init quickstart CLI and 20+ categorized example directories (function-calling, RAG, flows, multi-worker, observability, telephony) substantially lower the barrier to a working first agent, though the frame/processor abstraction itself has a real learning curve for developers new to streaming pipeline architectures.