redbeat

RedBeat replaces Celery Beat’s default file-based scheduler with one backed entirely by Redis. Instead of reading the whole schedule off disk at startup, RedBeat stores each task’s definition and metadata in a Redis hash and tracks due times in a sorted set used as a priority queue, so startup stays fast even with a large number of scheduled tasks and any language with a Redis client can create, modify, or remove tasks at runtime without restarting Beat.

It also adds operational safeguards Celery’s built-in scheduler lacks: a distributed lock (with a custom Lua script that extends to a fixed expiration rather than adding time) prevents two Beat processes from running against the same schedule, standby nodes can take over automatically if the lock holder dies, and a startup check detects Redis eviction policies that would silently drop schedule entries. RedBeat supports standalone Redis, TLS, Sentinel, and Redis Cluster, and is compatible with Valkey as a drop-in server.

What You Get

• A drop-in Scheduler class (redbeat.RedBeatScheduler) configured via beat_scheduler/-S redbeat.RedBeatScheduler
• Redis-backed schedule storage using a sorted set as a priority queue plus per-task hashes for definitions and metadata
• A distributed lock with a corrected Lua extend script so only one Beat instance runs at a time, with automatic failover to standby nodes
• Runtime task creation, modification, and deletion from any language with a Redis client — no Beat restart required
• Support for standalone Redis, TLS (rediss://), Sentinel, and Redis Cluster connection modes, plus Valkey server compatibility
• A startup check for dangerous maxmemory-policy eviction settings, configurable as ignore/warn/error/raise

Common Use Cases

• Dynamic task scheduling - applications that create or adjust periodic tasks at runtime (e.g. per-tenant cron jobs) instead of redeploying a static beat_schedule
• High-availability Celery Beat - running multiple Beat processes as hot standbys so a crashed node’s schedule is picked up automatically via the distributed lock
• Large schedules - deployments with thousands of periodic tasks where loading the full schedule into memory at startup would be slow
• Cross-language schedule management - external services or admin tools that add/remove scheduled tasks directly in Redis without going through Celery code

Under The Hood

Architecture RedBeat plugs into Celery’s scheduler extension point: RedBeatScheduler subclasses celery.beat.Scheduler and RedBeatSchedulerEntry subclasses celery.beat.ScheduleEntry, so Celery drives the same tick loop it always does while every read and write of schedule state goes through Redis instead of memory. The schedule itself lives in a sorted set keyed by each task’s next-due UNIX timestamp (acting as a priority queue), with the task’s JSON-encoded definition and run metadata stored in a companion Redis hash per entry (schedulers.py). Connection setup is centralized in get_redis()/RedBeatConfig, which resolves standalone, SSL, Sentinel, and Cluster Redis targets from Celery config and optionally wraps the connection in a RetryingConnection that uses tenacity to retry on connection/timeout errors. A distributed lock, acquired in the beat_init signal handler and renewed every tick(), guarantees only one Beat process is active; it patches redis-py’s default lock-extend behavior with a custom Lua script (LUA_EXTEND_TO_SCRIPT) so renewals extend to a fixed expiration rather than compounding. checks.py isolates a startup-only Redis maxmemory-policy check, and decoder.py/schedules.py handle JSON encoding of Celery schedule types including rrules. Because RedBeatSchedulerEntry calls get_redis(self.app) directly throughout rather than going through a shared repository, any change to connection resolution has to be traced through every read/write site individually.

Tech Stack A pure-Python 3.9+ package (celery-redbeat on PyPI, imported as redbeat) built with pbr/setuptools and configured entirely through setup.cfg. Runtime dependencies are celery>=5.0 (extends celery.beat.Scheduler and hooks the beat_init signal), redis>=3.2 (redis-py, including its Sentinel and RedisCluster clients), python-dateutil (recurrence-rule support in schedules.py), and tenacity (the connection retry wrapper). Style and linting run through black and isort (configured in pyproject.toml) and flake8 (in setup.cfg); GitHub Actions runs a five-version Python test matrix (3.9-3.13) plus a dedicated lint job, and mise pins the local dev Python version. There’s no web framework or database ORM involved — Redis is the only external dependency at runtime.

Code Quality The tests/ directory has real coverage: test_scheduler.py is the largest suite and exercises RedBeatScheduler/RedBeatSchedulerEntry save, delete, reschedule, and next-instance behavior, with dedicated suites for config resolution (test_config.py), the eviction check (test_checks.py), entry lifecycle (test_entry.py), JSON encode/decode round-tripping (test_json.py), and schedule types (test_schedules.py). Tests run via Python’s built-in unittest (python -m unittest discover tests or make test) with shared fixtures in basecase.py. Error handling favors explicit logging over silent failure: maybe_due() catches exceptions from apply_async and logs rather than crashing the beat loop, tick() narrowly catches RuntimeError during iteration, and the eviction-policy check surfaces findings at warn/error/raise severity depending on configuration rather than failing silently. Type hints are used sparingly (a few Any/str annotations) with no mypy or static type checking configured, so correctness leans on the test suite and runtime checks rather than a type system.

What Makes It Unique RedBeat’s core bet — a Redis sorted set as a due-time priority queue plus per-task hashes — means Beat’s startup cost and per-tick scan don’t scale with total schedule size the way a fully in-memory schedule does, and it lets any process with Redis access create or modify scheduled tasks without touching Celery code or restarting Beat. Its distributed lock (with the corrected Lua extend-to-timeout script) is a deliberate fix for a known bug in the naive lock-extension approach, letting multiple Beat nodes run as hot standbys with automatic failover instead of requiring a single, manually managed instance. The startup maxmemory-policy check is a small but pointed addition: because RedBeat’s Redis keys are the schedule (not a cache of it), it proactively warns when a deployment’s eviction policy could silently drop scheduled tasks — an operational failure mode most schedulers don’t check for at all.