MLX-VLM

MLX-VLM is a Python package for running inference and fine-tuning Vision Language Models (VLMs) — plus omni models that add audio and video understanding — natively on Apple Silicon using Apple’s MLX framework. It ships a Python API, a CLI (mlx_vlm.generate, mlx_vlm.server, mlx_vlm.chat_ui), and an OpenAI/Anthropic-compatible FastAPI server, giving developers a single toolkit to load, quantize, batch-generate, and LoRA/DoRA fine-tune over 160 vision-language model architectures directly on Mac hardware without a discrete GPU.

Beyond basic generation, the library implements production-grade inference features rarely found together in one package — speculative decoding (DFlash, EAGLE-3, MTP drafters), automatic prefix caching, KV-cache quantization, vision feature caching, and distributed inference — making it as suited to local experimentation as to serving multimodal models at scale on Apple hardware.

What You Get

• Python API (load, generate, stream_generate, batch_generate) plus CLI entry points (mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.convert, mlx_vlm.server, mlx_vlm.chat_ui)
• Support for 160+ vision-language and omni model architectures (Qwen2-VL, LLaVA, Idefics, PaliGemma, Florence-2, Molmo, Pixtral, DeepSeek-OCR, and more)
• An OpenAI- and Anthropic-compatible FastAPI server with continuous batching, automatic prefix caching, and KV-cache quantization
• LoRA/DoRA fine-tuning and SFT/ORPO trainers for adapting VLMs on local Apple hardware
• Speculative decoding support via DFlash, EAGLE-3, and MTP drafter architectures for 2-4x faster generation

Common Use Cases

• Running local multimodal chat or a Gradio UI against a VLM entirely offline on a Mac
• Serving a VLM behind an OpenAI/Anthropic-compatible API for existing client tooling
• Fine-tuning a vision-language model with LoRA/DoRA on a custom image-text dataset
• Extracting structured data (OCR, object detection, document understanding) from images at the CLI or via the Python API
• Benchmarking or research on speculative decoding, prefix caching, and quantization strategies for multimodal LLMs

Under The Hood

Architecture MLX-VLM organizes around a plugin-style model registry under mlx_vlm/models/ (168 subpackages, one per architecture family — Qwen2-VL, LLaVA, PaliGemma, DeepSeek-OCR, Moondream, etc.), each implementing a common interface defined in models/base.py so that utils.py’s load() can dynamically dispatch to the right config/model/processor classes based on the Hugging Face config’s model_type. Generation itself is centralized in the generate/ package (ar.py for autoregressive decoding, diffusion.py for diffusion-model VLMs, dispatch.py routing between them), while server/app.py layers a FastAPI application with OpenAI- and Anthropic-shaped request/response schemas (openai.py, anthropic.py) on top of the same generation core, and trainer/ adds LoRA/DoRA adapter injection (lora_layers.py, dora_layers.py) plus SFT/ORPO training loops reusing the same model abstractions.

Tech Stack Built on Apple’s mlx array/ops library (>=0.32.0) and mlx-lm/mlx-audio for text/audio backbones, with transformers (>=5.14.0) used for tokenizers/processors and Hugging Face config compatibility, Pillow and opencv-python for image preprocessing, fastapi/uvicorn/starlette for the server, and llguidance for constrained/structured generation. Packaging uses setuptools with dynamic versioning from mlx_vlm/version.py; requires-python >=3.10; optional extras split out gradio (ui), datasets (train), and CUDA/CPU MLX backends.

Code Quality The mlx_vlm/tests/ directory holds 47 test modules covering individual model families (test_bonsai.py, test_flux2.py, test_paddleocr_vl_vision.py), core mechanics (test_kv_cache_quantization.py, test_apc_*.py for automatic prefix caching, test_speculative.py, test_generate.py), and utilities (test_processors.py, test_prompt_utils.py); a tests.yml GitHub Actions workflow runs them on push. Code style is enforced via pre-commit hooks (black, isort —profile=black, autoflake for unused-import removal), and naming is consistent (snake_case modules mirroring each model’s Hugging Face family name).

API Design The public surface is deliberately small: from mlx_vlm import load, generate covers the common case, with stream_generate/batch_generate for streaming and batched workloads, and equivalent CLI commands (mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.convert, mlx_vlm.server, mlx_vlm.chat_ui) registered as console-script entry points so no Python is required to get started. The FastAPI server mirrors OpenAI’s and Anthropic’s request/response shapes directly, so existing SDKs and tooling built against those APIs work against a local MLX-VLM server with just a base-URL change — a notably low-boilerplate integration path for a self-hosted inference server.