django-q2

Django Q2 is a fork of the original Django Q, picking up development after the upstream project stalled in mid-2021. It runs background and scheduled work for Django applications using a multiprocessing worker pool rather than requiring an external broker daemon like Celery does, though it also supports Redis, Amazon SQS, IronMQ, and MongoDB as brokers alongside a zero-dependency Django ORM broker.

Tasks are queued with a single async_task() call and results, failures, and hooks are tracked through regular Django models, so they show up in the Django admin without extra tooling. Scheduling supports one-off, repeated, and full cron expressions via Schedule objects that are just database rows, making them easy to create, inspect, and manage from code or the admin UI. Beyond the core queue/scheduler, it ships a cluster monitor, memory monitor, and signed/compressed task packages for safer serialization across processes.

What You Get

• A qcluster management command that starts a multiprocessing worker pool, sentinel, and monitor process
• async_task() and schedule() APIs for firing off background work and cron/interval schedules from application code
• Choice of broker: Redis, Amazon SQS, IronMQ, MongoDB, or a dependency-free Django ORM broker
• Django admin integration for inspecting queued, successful, and failed tasks and managing schedules
• Result hooks, task groups, and task chains for composing multi-step background workflows
• Signed and optionally compressed task packages, plus built-in Sentry and Rollbar failure reporting hooks

Common Use Cases

• Offloading slow operations (emails, report generation, image processing) from the request/response cycle in a Django app
• Running recurring jobs — daily digests, cache warm-ups, cleanup tasks — via cron-style or interval schedules stored as Django models
• Adding background processing to a Django project without standing up a separate Celery + broker deployment
• Chaining multi-step async workflows (fetch, transform, notify) with task chains and completion hooks
• Deploying on PaaS platforms across multiple instances using Redis, SQS, or MongoDB as a shared broker

Under The Hood

Architecture Execution flows from the qcluster management command into a Cluster object (django_q/cluster.py) that spins up a Sentinel process supervising a multiprocessing worker pool, a pusher that pulls tasks off the configured broker and feeds an internal Queue, and a monitor process that reports status via django_q.monitor. Tasks and schedules are modeled as plain Django models (django_q/models.py), so the ORM itself doubles as both persistence and, optionally, the message broker (django_q/brokers/orm.py) — a deliberate simplification versus systems that require a separate broker service. The broker layer is abstracted behind a common Broker interface (django_q/brokers/init.py) with concrete implementations for Redis, SQS, IronMQ, and MongoDB, so swapping transport is a configuration change rather than a code change; the worker pool then deserializes signed packages (django_q/signing.py) and executes the target function, writing results back through the same model layer the admin reads from.

Tech Stack The project is pure Python targeting Django 5.2 through 6.1, with django-picklefield as its only hard dependency for storing arbitrary task arguments/results on models. Everything broker-specific is an optional extra declared in pyproject.toml — django-redis/hiredis for Redis, boto3 for SQS, iron-mq for IronMQ, pymongo for MongoDB — so a self-hosted deployment only installs what it needs. Build tooling is uv-based (uv_build), linting runs through Ruff (ruff.toml enabling import-sorting), and dependency locking uses uv.lock; a Dockerfile and docker-compose files support both a runnable example project and a Dockerized test matrix.

Code Quality Tests live under django_q/tests/ and cover the cluster, brokers, scheduler, admin, and management commands as discrete modules, executed via pytest with a dedicated Django settings module (pytest.ini) — a conventional, comprehensive layout for a Django-ecosystem package. Error handling favors explicit, narrow except clauses (e.g. catching AppRegistryNotReady during early import, or psutil.NoSuchProcess when polling worker memory) rather than broad catch-alls, and public methods use type hints where the return type isn’t obviously implied (-> int, -> bool, -> str on the broker interface). CI runs the test suite plus CodeQL static analysis on every push and PR, and a separate release workflow automates publishing on tagged versions.

API Design The public surface is intentionally small: async_task() for fire-and-forget work and schedule() for recurring jobs, both accepting a dotted function path or callable plus arbitrary args/kwargs, with optional keys (hook, group, chain, cached, sync, timeout) layered on top rather than forcing configuration up front. Because schedules and results are just Django models, there’s no separate query language or client to learn — anything already familiar with Django’s ORM and admin can inspect and manage background work directly, and the extensive Sphinx documentation (docs/, covering architecture, brokers, chains, and schedules individually) keeps the learning curve low despite the range of broker and scheduling options.