Whisper

Whisper is OpenAI’s open-source automatic speech recognition (ASR) library. A single Transformer sequence-to-sequence model handles multilingual transcription, speech-to-English translation, spoken language identification, and voice activity detection, replacing what was traditionally a multi-stage speech pipeline.

The package ships six model sizes from tiny (39M parameters) to large (1550M), plus a turbo model optimized for fast English transcription, so users can trade off accuracy against memory and inference speed. Models are downloaded on first use and cached locally, and everything runs offline once fetched.

Whisper is usable both as a command-line tool (whisper audio.mp3 --model turbo) and as a Python library (import whisper; model = whisper.load_model("turbo")), with lower-level access to language detection and decoding for applications that need more control than the high-level transcribe() call.

What You Get

• Six model sizes (tiny through large) plus a turbo variant, each with English-only and multilingual checkpoints, covering a range of speed/accuracy/VRAM tradeoffs
• A command-line whisper tool for transcribing or translating audio files directly from the terminal
• A Python API (whisper.load_model, model.transcribe) for embedding speech recognition into applications
• Lower-level primitives (load_audio, log_mel_spectrogram, detect_language, decode) for building custom transcription pipelines
• Word-level timestamp support via cross-attention alignment heads baked into each model checkpoint
• Automatic checksum-verified model downloading and local caching under ~/.cache/whisper

Common Use Cases

• Transcribing podcasts, meetings, or interviews into searchable text
• Translating non-English speech into English subtitles or transcripts
• Adding voice input or dictation to an application without relying on a hosted API
• Building automatic subtitle/caption generation pipelines for video content
• Language identification as a preprocessing step in multilingual audio pipelines

Under The Hood

Architecture Whisper is a standard encoder-decoder Transformer (whisper/model.py): a convolutional audio encoder feeds sinusoidal-positioned frames into stacked ResidualAttentionBlocks, and a text decoder attends over both its own tokens and the encoder output via MultiHeadAttention, using PyTorch’s scaled_dot_product_attention where available with a manual fallback. Multitask behavior (transcribe vs. translate vs. detect-language) is expressed entirely as special tokens the decoder predicts, avoiding separate task-specific heads. transcribe.py orchestrates the higher-level pipeline — chunking audio into 30-second windows, running autoregressive decoding per window via decoding.py, and using precomputed per-model cross-attention “alignment heads” (timing.py) to derive word-level timestamps through dynamic time warping. Swapping the core Transformer block would ripple through decoding, timing, and every cached model checkpoint, since checkpoints encode both weights and the alignment-head metadata.

Tech Stack Pure Python on top of PyTorch, with numpy for array work, numba for JIT-accelerated DTW cost computation, tiktoken (OpenAI’s own tokenizer) for BPE tokenization, more-itertools for iteration helpers, and an optional Triton kernel (triton_ops.py) for GPU-accelerated median filtering. Packaging is PEP 621-based (pyproject.toml, setuptools backend, dynamic version from whisper/version.py) with a console-script entry point (scripts.whisper = whisper.transcribe:cli). Model weights are hosted on Azure blob storage and fetched by SHA256-verified URL rather than bundled or pulled from a model hub.

Code Quality Tests live under tests/ using pytest, covering model loading and transcription accuracy against a fixed sample clip, tokenizer round-tripping, audio loading, and DTW-based timing. CI (.github/workflows/test.yml) runs pre-commit (black, isort, flake8) plus the pytest suite across supported Python versions on every push and PR. Code is typed with dataclasses and standard typing annotations for public function signatures, though internals lean on implicit tensor shapes rather than strict typing throughout. Error handling is direct — model loading raises explicit RuntimeErrors for unknown model names or checksum mismatches rather than silently falling back.

What Makes It Unique Whisper’s distinguishing choice is training a single model, with no per-task fine-tuning, entirely through multitask token sequences on a very large and weakly-supervised (i.e., unfiltered web-scale) audio-text dataset. That lets one checkpoint replace a traditional pipeline of separate voice-activity-detection, language-ID, acoustic, and translation models, and gives it broad real-world robustness (accents, background noise, technical language) that earlier narrowly-trained ASR systems lacked. The turbo variant further demonstrates that this multitask model can be distilled down for near real-time use while retaining most of the accuracy of the largest checkpoint.