fish-speech

Fish Speech (Fish Audio S2) is an open-source, state-of-the-art multilingual text-to-speech system built around a Dual-Autoregressive (Dual-AR) transformer architecture: a large “slow” AR model predicts the primary semantic codebook while a much smaller “fast” AR head reconstructs the residual RVQ audio codec tokens at each step, trading a bit of architectural complexity for major gains in inference speed and audio fidelity. The model was trained on a large multilingual audio corpus and post-trained with Group Relative Policy Optimization (GRPO) reinforcement learning, using the model suite itself as a reward model to align generations with human preference across semantic accuracy, instruction adherence, and timbre similarity.

Beyond raw synthesis quality, Fish Speech supports fine-grained inline control of prosody and emotion through free-form natural-language tags (not a fixed preset list), native multi-speaker generation via per-speaker conditioning tokens, multi-turn dialogue-aware generation, and rapid voice cloning from short reference clips. Because the Dual-AR architecture is structurally similar to a standard decoder-only LLM, the project can be served through SGLang with continuous batching, paged KV cache, and RadixAttention prefix caching for low-latency, high-throughput deployment.

The repository ships a command-line inference tool, a Gradio WebUI, and a production-style API server (Kui/uvicorn, msgpack request format, bearer-token auth) with Docker Compose files for both CUDA and ROCm targets, plus training and LoRA fine-tuning scripts built on PyTorch Lightning and Hydra. The codebase itself is released under the custom Fish Audio Research License, which permits self-hosted use for research and non-commercial purposes but requires a separate paid license for any commercial deployment.

What You Get

• Pretrained Fish Audio S2 Pro model weights distributed via HuggingFace for direct inference
• A command-line inference pipeline plus an interactive Gradio WebUI for quick experimentation
• A production-style API server (Kui + uvicorn, msgpack payloads, bearer-token auth, OpenAPI docs) ready for Docker deployment
• Docker Compose configurations for both CUDA and AMD ROCm GPU targets
• Training and LoRA fine-tuning scripts built on PyTorch Lightning and Hydra-composed configs
• A companion TypeScript/React WebUI package (awesome_webui) for a richer front-end experience

Common Use Cases

• Cloning a speaker’s voice from a short reference sample for narration or dubbing
• Adding multilingual text-to-speech to an application without depending on a single closed-source vendor
• Generating expressive audiobook or video narration using inline emotion/prosody tags
• Building low-latency, high-throughput voice agents served through SGLang
• Self-hosting voice synthesis so customer audio and text never leave your own infrastructure

Under The Hood

Architecture The project separates a model core (fish_speech/) from its serving tools (tools/). The core defines the Dual-AR transformer and its variants (naive, dual_ar, and a fish-qwen3-omni flavor) as dataclasses loaded from checkpoint JSON via a from_pretrained dispatch in llama.py, alongside the residual-codec model (models/dac), tokenizer, and content-sequence/conversation builders that assemble prompts into token streams. The serving layer builds on top: tools/api_server.py wires a ModelManager singleton into a Kui ASGI app with bearer-token middleware and msgpack request handling, tools/webui provides a Gradio front end, and tools/llama/tools/vqgan hold offline utilities for dataset building, LoRA merging, and codec extraction. Because the model-config schema is shared across the core transformer, the inference engine, and the server layer, changes to it ripple through all three — a fairly standard but tightly coupled research-to-production ML repo layout.

Tech Stack Built on Python with PyTorch and torchaudio, HuggingFace transformers for tokenization utilities, PyTorch Lightning for training orchestration, and Hydra for composable configs. The inference API server runs on the Kui ASGI framework with uvicorn, ormsgpack for compact request/response payloads, and CORS support; the WebUI runs on Gradio, with an additional TypeScript/React front end maintained separately. Audio processing draws on librosa, resampy, and a Descript Audio Codec-based neural codec, with Silero VAD for voice-activity detection. Dependencies are locked with uv, and deployment targets both CUDA and AMD ROCm via separate Docker Compose files.

Code Quality Automated test coverage is limited to a single focused unit-test module covering KV-cache attention behavior with mocked models; there is no broader test suite for the inference, server, or training code paths, and no CI workflow runs tests (the GitHub Actions in this repo build Docker images, publish docs, and manage stale issues). Pre-commit hooks enforce isort and black formatting plus basic file hygiene, and a Pyright config is present, but neither a linter nor type-checking is enforced in CI. Type hints and dataclasses are used throughout the model code, and loguru-based structured logging with a dedicated exception handler backs the API layer, but there’s no CONTRIBUTING guide to codify these conventions for outside contributors.

What Makes It Unique The Dual-AR design — a large autoregressive model for primary semantic tokens paired with a lightweight secondary AR head for residual codec tokens — is a genuine architectural choice rather than a wrapper around an existing diffusion or single-pass TTS model, and it’s paired with GRPO-based reinforcement learning alignment using the model suite itself as its own reward model. Because the resulting architecture is structurally close to a standard decoder-only LLM, the project can reuse LLM-serving techniques (continuous batching, paged KV cache, prefix caching) via SGLang for extremely low first-audio latency, and it layers on free-form inline emotion/prosody tags and native multi-speaker, multi-turn conditioning that go beyond the fixed-style-preset approach common in other open TTS projects.

Self-Hosting

Licensing Model Released under the custom Fish Audio Research License rather than an OSI-approved open-source license: self-hosting and modification are free for research and non-commercial purposes only, and any commercial use requires a separate paid license from Fish Audio.

Self-Hosting Restrictions

• Self-hosting is permitted without a paid license only for research or non-commercial purposes (personal/hobbyist use, evaluation and testing, academic research).
• Any commercial purpose — building a paid product or hosted API on top of it, or using it in a business’s internal operations — requires contacting Fish Audio for a commercial license.
• Distributed copies or derivative works must retain the license’s required attribution notice and display “Built with Fish Audio” per the license text.

Enterprise Features There is no feature-gated open-source/paid split in the code itself — the restriction is purely licensing-based (a commercial-use fee), not a technical limitation enforced by the software.

Cloud vs Self-Hosted Not applicable: the same codebase and model weights are used for both non-commercial self-hosting and commercial deployment; no separate cloud-only feature set was found in this repository.

License Key Required No. The code does not technically enforce or check for a commercial license key — the commercial-use restriction is a legal/contractual obligation under the Fish Audio Research License, not something blocked in-app.