conformer

conformer is a compact PyTorch library implementing the convolution module and full encoder block from the “Conformer: Convolution-augmented Transformer for Speech Recognition” paper. Rather than reimplementing an entire ASR training pipeline, it exposes three drop-in nn.Module classes — a standalone depthwise-separable conv block, a full macaron-style Conformer block combining feedforward, relative-position attention, and convolution, and a stack of those blocks — so researchers can plug conv-augmented attention into any sequence model.

The implementation targets exactly one thing: giving Transformer-style architectures the local inductive bias that pure self-attention lacks, using the same GLU-gated depthwise convolution design as the original paper, plus support for causal (streaming) variants. It is maintained by Phil Wang (lucidrains), whose single-file, minimal-dependency style makes the module easy to read, audit, and adapt for other sequence modeling tasks beyond speech.

What You Get

• ConformerConvModule — the paper’s depthwise-separable convolutional block (LayerNorm, pointwise conv, GLU, depthwise conv, BatchNorm, Swish, pointwise conv) as a single nn.Module
• ConformerBlock — the macaron-style block combining two half-step feedforward layers, relative-position multi-head attention, and the conv module with residual connections and a final LayerNorm
• Conformer — a full encoder that stacks depth copies of ConformerBlock for direct use as a sequence encoder
• Causal mode (causal=True) that swaps the conv module’s padding and normalization for streaming/auto-regressive use
• Shaw-style learned relative positional embeddings built directly into the attention layer, so no separate positional encoding module is needed

Common Use Cases

• Speech recognition model builders assembling ASR encoders can drop in ConformerConvModule or the full Conformer stack instead of reimplementing the paper’s conv-augmented transformer block
• Sequence modeling researchers experimenting with combining local convolutional receptive fields and global self-attention can reuse this block as a component in a larger architecture
• Streaming/causal audio applications can use the causal=True option to build online inference pipelines where future context isn’t available
• Anyone studying the Conformer paper can use this compact, single-file implementation as a readable reference instead of tracing the module through a full training framework

Under The Hood

Architecture Everything lives in one file (conformer/conformer.py, ~250 lines) re-exported through init.py as three public classes. The design follows a clean layered nn.Module hierarchy: Conformer (a ModuleList of depth blocks) wraps ConformerBlock (two Scale/PreNorm-wrapped FeedForward layers plus a PreNorm-wrapped Attention and a ConformerConvModule, combined with hand-written residual adds), which in turn is built from leaf modules (FeedForward, Attention, ConformerConvModule) composed from primitives (Swish, GLU, DepthWiseConv1d). There’s no configuration system, dependency injection, or I/O layer — the load-bearing abstraction is the shared dim argument threaded through every submodule, and the Rearrange calls in ConformerConvModule encode a strict b n c ↔ b c n tensor-layout contract that any structural change would need to preserve.

Tech Stack The only runtime dependencies are torch and einops (>=0.6.1), used for rearrange/Rearrange in the attention and conv modules. Packaging is a plain setup.py with setuptools.find_packages() — no pyproject.toml, no lockfile, no CLI. The single GitHub Actions workflow (python-publish.yml) only builds and twine-uploads a release to PyPI; there is no test or lint job. There is no deployment target: the package is meant to be pip-installed and imported into a larger PyTorch training or inference codebase.

Code Quality No test files exist anywhere in the repository. Error handling is minimal — the code relies on small exists/default helpers rather than explicit validation, and shape mismatches would surface as raw PyTorch tensor errors. Naming is consistent with lucidrains’ other repos and with common attention-model conventions (dim, heads, dropout), but there are no type hints, no linter or formatter configuration, and CI does not run tests or static checks of any kind.

API Design The public surface is deliberately tiny: three classes covering three levels of granularity (conv module, block, full stack), each with keyword-only constructor arguments and sensible defaults matching the paper’s reported settings. Getting started requires no more than importing a class and calling it on a tensor, as shown directly in the README for all three levels. There are no docstrings and no external docs site — the README’s runnable code samples are the entire documentation surface — but the consistent kwarg style across lucidrains’ ecosystem of similar packages makes it easy to move between them.