concurrent-log-handler

concurrent-log-handler provides robust logging handlers for Python’s standard logging package, built for applications that run as multiple processes (and threads) — potentially spread across hosts sharing a network drive — that all need to write to a single, centralized log file without dropping records or corrupting output during rotation.

It ships two drop-in handlers, ConcurrentRotatingFileHandler and ConcurrentTimedRotatingFileHandler, that extend the stdlib’s own rotating handlers rather than replacing them outright. Concurrency safety comes from portalocker-based file locking around every write and rotation, atomic file creation with correct permissions to close a permission-race window, and fork-awareness so child processes created via os.fork() don’t silently share a broken lock state. Rotation supports both size-based and time-based triggers with optional gzip compression, and works across Windows and POSIX.

What You Get

• ConcurrentRotatingFileHandler, a size-based rotating handler safe for concurrent multi-process writes
• ConcurrentTimedRotatingFileHandler, a time-based rotating handler with the same cross-process safety guarantees
• Cross-platform advisory file locking via portalocker, covering both POSIX and Windows
• Optional gzip compression of rotated log files with configurable permissions and ownership
• Fork-safety: handlers inherited across os.fork() automatically reopen their lock state instead of corrupting it

Common Use Cases

• Multiple worker processes (Gunicorn, uWSGI, multiprocessing pools) logging to one shared file
• Applications on several hosts writing to a centralized log file over a shared network drive
• Replacing RotatingFileHandler in an existing app to stop losing log messages when rotation fails on Windows
• Long-running services that need size- or time-based log rotation with compression and no external logging server

Under The Hood

Architecture The package is a single, tightly-scoped module (src/concurrent_log_handler/init.py, roughly 1,500 lines) exposing two classes: ConcurrentRotatingFileHandler, which extends logging.handlers.BaseRotatingHandler, and ConcurrentTimedRotatingFileHandler, which extends TimedRotatingFileHandler. Rather than reimplementing logging.Handler, both subclass the stdlib’s own rotating handlers so they work as drop-in replacements. Concurrency safety is centered on portalocker-based advisory locking (_do_lock/_do_unlock, _open_lockfile) wrapped around every write and rotation, atomic file creation with pre-set permissions (_atomic_create_with_perms) to close a race window between file creation and chmod, and umask handling (_alter_umask) for permission-safe writes across processes with different users. Rotation (doRollover, do_rename, do_gzip) is carefully sequenced to avoid one process clobbering a file mid-rotation by a sibling process, and rollover timestamps are persisted to disk (read_rollover_time/write_rollover_time) so time-based rotation survives process restarts. There’s no plugin system or dependency injection — it’s a deliberately narrow module, though both classes hook deep into stdlib internals (_open, emit, shouldRollover), so a stdlib logging change would require updates on both sides.

Tech Stack Pure Python supporting 3.6 through 3.14, with a single runtime dependency on portalocker (>=2.6.0) for cross-platform advisory file locking; on Windows, portalocker in turn depends on pywin32. Packaging uses Hatchling as the build backend, with the version sourced dynamically from a dedicated version.py file. The dev/test toolchain — pytest with pytest-mock, pytest-sugar, pytest-cov, and pytest-repeat, plus coverage.py, black, ruff (a large enabled rule set spanning bandit-style security checks, bugbear, complexity, and pylint subsets), and mypy in a fairly strict configuration — is orchestrated through Hatch-managed environments. GitHub Actions runs the test suite in CI. There’s no web framework, ORM, or database involved; this is a standalone logging utility, not an application.

Code Quality The test suite specifically targets the hard edges of this domain: multi-process rotation, permission-race windows, interpreter-shutdown logging, and timed-rollover edge cases each get a dedicated test module, alongside stress-test and benchmark scripts for performance validation. Error handling favors explicit errno inspection and OSError subclassing over bare excepts, with contextlib.suppress used narrowly rather than as a catch-all. Type hints are used consistently throughout (Optional, Union, Tuple, Dict from typing), the package ships a py.typed marker for downstream type-checkers, and mypy runs in a strict-leaning configuration (disallow_untyped_defs, warn_return_any, and similar). Naming mirrors the stdlib logging handlers being extended (doRollover, shouldRollover, emit), and CI enforces both the test suite and ruff/black formatting.

What Makes It Unique The project doesn’t introduce a new logging paradigm — it’s a targeted fix for two well-known, specific gaps in stdlib logging: RotatingFileHandler dropping records when rotation fails (particularly on Windows), and the stdlib handlers being unsafe when multiple OS processes share one log file. Its concrete contribution is flock-based cross-process serialization (via portalocker) around every write and rotation, atomic pre-created files with correct permissions to eliminate a permission-race window, and explicit fork-awareness so handlers inherited by forked child processes reopen their lock state instead of silently breaking it. These are narrow, documented production pain points rather than a new architecture — a standard technique (file locking) applied with unusual care to a specific, well-understood defect class.