celery-once

Celery Once is a small extension for Celery that stops a task from being queued or run more than once at the same time. It works by providing a QueueOnce abstract base task that tasks can inherit from: behind the scenes it overrides apply_async/delay to check a distributed lock before scheduling, raising an AlreadyQueued exception (or returning None in graceful mode) if a matching task is already in flight.

Locks are keyed by the task’s name plus its bound arguments (or a restricted subset of them via the keys option), so calls with different arguments are tracked independently. Locking itself is pluggable — a Redis backend built on redis-py’s distributed Lock primitive is the default and recommended choice for production, while a file-based backend using atomic O_CREAT|O_EXCL file creation is available for simpler, single-machine setups. Both backends support a configurable timeout so stale locks expire automatically even if a task fails without releasing its lock.

What You Get

• A QueueOnce abstract Celery task base class that transparently wraps apply_async/delay with a distributed lock check
• An AlreadyQueued exception raised when a duplicate task is already queued or running, with an optional graceful mode that returns None instead
• Argument-aware lock keys, with the ability to restrict locking to a subset of a task’s arguments via once={'keys': [...]}
• A pluggable backend system shipping with Redis (via redis-py’s Lock) and File (atomic O_CREAT|O_EXCL) implementations, plus a documented interface for writing custom backends
• Configurable per-task or per-call lock timeouts, and an unlock_before_run option to release the lock before execution instead of after

Common Use Cases

• Preventing a scheduled/periodic task from overlapping with itself if a previous run is still in progress
• De-duplicating tasks triggered by webhooks or events that can fire multiple times for the same underlying operation
• Serializing access to a task that touches a resource (e.g. a specific database row or external API) that shouldn’t be modified concurrently
• Coalescing bursts of identical task calls (e.g. cache-rebuild or notification tasks) into a single in-flight execution

Under The Hood

Architecture Celery Once is a small, single-purpose extension rather than a layered application: celery_once/tasks.py defines the QueueOnce abstract Task subclass that overrides apply_async (to check-and-acquire a lock before scheduling), __call__ (for the unlock_before_run variant), and after_return (to release the lock once the task completes or fails). Lock keys are generated by celery_once/helpers.py, which binds the task’s actual call arguments via inspect.signature and serializes them alongside the task name into a deterministic string, optionally restricted to a subset of argument names. Locking itself is delegated to a pluggable backend resolved at runtime from the Celery app’s ONCE config through import_backend, keeping the task-wrapping logic decoupled from the storage mechanism. It integrates purely through Celery’s task inheritance model — no separate service, process, or app scaffolding is introduced.

Tech Stack Pure Python with two runtime dependencies: celery (the task framework it augments) and redis>=2.10.2 (used by the default locking backend); six and a funcsigs fallback handle Python 2/3 compatibility for the now-legacy Python 2.7/3.3-3.5 support declared in setup.py. There is no modern pyproject.toml — packaging is a classic setup.py/MANIFEST.in setup. Testing infrastructure includes tox.ini for multi-version test runs, .travis.yml for CI, and a docker-compose.yml/Dockerfile pair used to spin up a real Redis instance for the integration test suite.

Code Quality The project has both unit tests (tests/unit, covering the key-generation helpers) and integration tests (tests/integration, including per-backend suites and a Flask app fixture for testing app-context integration), run with pytest and configured via pytest.ini to skip framework-specific tests by default. Error handling is explicit — failures surface as a single well-defined AlreadyQueued exception rather than being swallowed — and naming throughout is clear and idiomatic. The codebase predates type hints and carries no linter/formatter configuration, and while Travis CI and a Coveralls badge are wired up, the project has seen no recent commits and no migration to newer CI tooling.

What Makes It Unique The core idea — a distributed lock wrapped around task scheduling — is a well-established pattern in job-queue ecosystems rather than a novel invention. Celery Once’s value is in its ergonomics: a pluggable backend abstraction that lets you swap Redis for a local file lock without touching task code, argument-aware keys that can be scoped to a subset of a task’s parameters, and a graceful mode that turns a would-be exception into a quiet no-op. These are practical conveniences rather than groundbreaking technical choices.