huey

Huey is a small, focused task queue for Python written by Charles Leifer (also the author of Peewee). It exposes a decorator-based API — @huey.task() and @huey.periodic_task() — so any function can be turned into an asynchronously-executed, retryable, schedulable unit of work with almost no boilerplate. A separate huey_consumer process picks up enqueued tasks and runs them using one of several worker models (multi-process, multi-thread, or gevent greenlets).

What sets Huey apart from heavier alternatives is its storage flexibility: the same task-decorator API works unchanged against Redis (and Redis-compatible stores like Valkey/Dragonfly), Postgres, SQLite, the filesystem, or pure in-memory storage, letting a project start with zero extra infrastructure and move to Redis later without touching task code. It also ships native Django integration (including an admin panel for task visibility) and Flask-Peewee admin support, plus first-class primitives for cron-style periodic tasks, automatic retries with backoff, task locking, rate limiting, timeouts, pipelines/chains, and fan-out/fan-in groups and chords.

What You Get

• Decorator-based task API (@huey.task(), @huey.periodic_task()) requiring no separate task registration step
• Five interchangeable storage backends — Redis, Postgres, SQLite, filesystem, and in-memory — behind one consistent interface
• A huey_consumer CLI that runs workers as processes, threads, or gevent greenlets, with health checks and graceful shutdown
• Built-in scheduling: run-at, run-after-delay, and crontab-style periodic tasks
• Automatic retries with configurable delay/backoff, task timeouts, locking, and rate limiting
• Task pipelines/chains, fan-out groups, and map-reduce chords for composing multi-step workflows
• Native Django integration (including an admin panel) and Flask-Peewee admin support

Common Use Cases

• Offloading slow operations (email sending, image processing, webhooks) from a Django or Flask request/response cycle
• Running nightly or cron-style maintenance jobs (backups, report generation, cache warming) via @huey.periodic_task(crontab(...))
• Building a lightweight background-job system for a small-to-medium app without standing up a full Celery + broker deployment
• Composing multi-step async workflows with retries and locking, such as fetch-then-process-then-notify pipelines
• Adding task queueing to an app that already uses SQLite or Postgres, without introducing Redis as a new dependency

Under The Hood

Architecture Huey separates the public API (api.py: the Huey class, TaskWrapper, Task) from storage backends (storage.py: BaseStorage subclasses for Memory, Redis, Sqlite, Postgres, File, and BlackHole) and the consumer process (consumer.py: Worker, scheduler, and process/thread/greenlet orchestration). Tasks are serialized via registry.py’s Message namedtuple plus a pickle-based Serializer, enqueued into the storage backend’s queue, then dequeued by the Consumer, reconstituted into a Task via Registry.create_task, and executed by a Worker of the configured type — with lifecycle hooks published through signals.py. Chains and chords are modeled by embedding on_complete/on_error/chord_config sub-messages recursively inside Message, letting the result store drive fan-in without a separate coordinator process. The real architectural seam is the storage interface: roughly a dozen storage subclasses share one BaseStorage contract, and swapping the core Task/Registry abstraction would ripple through every backend’s serialize/deserialize path as well as the Django and Flask-Peewee contrib integrations.

Tech Stack The core is pure Python 3 standard library (logging, threading, multiprocessing, optparse) with zero hard dependencies. Optional extras are declared in pyproject.toml: redis-py for Redis/Valkey-compatible brokers, psycopg for Postgres, and cysqlite for a faster SQLite driver, alongside stdlib sqlite3 for the default SqliteHuey. The huey_consumer console-script entry point is built on optparse rather than argparse or click, reflecting the project’s long history (started 2011). Storage code speaks each backend’s native protocol directly — raw Redis commands via redis-py, raw SQL via sqlite3/psycopg — with no ORM layer in between, and packaging uses setuptools with the version pulled dynamically from huey.__version__.

Code Quality The repo carries 17 dedicated test modules under huey/tests/ (test_api.py, test_storage.py, test_consumer.py, test_priority.py, test_django_tasks.py, test_postgres.py, and more), covering each storage backend, the consumer/worker loop, crontab scheduling, chords, and Django task-backend integration, run through a custom runtests.py. No GitHub Actions workflow was found in the cloned tree, so test execution looks manual/release-time rather than automated per pull request. Error handling is exception-based with a dedicated exceptions module (TaskException, RetryTask, TaskLockedException, ResultTimeout) rather than silent failures, and public classes carry descriptive docstrings, but there are no type hints, mypy configuration, or linter configuration in the repo.

API Design The decorator-first API (@huey.task(), @huey.periodic_task(crontab(...))) needs almost no boilerplate — importing one Huey subclass and decorating a function is enough to get an enqueueable task, and calling the decorated function returns a Result handle immediately with res() to block for the value. Naming stays consistent across the roughly ten Huey subclasses (RedisHuey, PriorityRedisHuey, SqliteHuey, CySqliteHuey, PostgresHuey, FileHuey, MemoryHuey, BlackHoleHuey), so switching storage backends is typically a one-line change. Extensive Sphinx documentation (guide, API reference, consumer, Django integration) plus five runnable example apps (simple, Django, Flask, mini, deploy) keep the ramp-up cost low.