smmap

smmap is a pure-Python sliding window memory-map manager. It wraps an interface around mmap and tracks mapped files along with how many clients use each map, automatically unloading unused regions when the system runs low on file descriptors or hits a configured memory limit.

By mapping only portions of a file at a time and unloading least-recently-used regions on demand, smmap makes it possible to process very large files even on 32-bit systems. It underpins GitPython’s object database and offers both a low-level native interface and a simpler string-like Buffer abstraction.

What You Get

• A memory manager that maps only portions of files and unloads unused regions
• LRU-based automatic unloading when file-descriptor or memory limits are hit
• A string-like Buffer interface hiding the low-level map details
• A simplified full-file mapping mode optimized for 64-bit applications
• Cross-platform support on Linux, macOS, and Windows for Python 3.7+

Common Use Cases

• Reading many large files with random-access patterns efficiently
• Processing files larger than addressable memory on 32-bit systems
• Backing an object database that memory-maps pack files (as in GitPython)
• Bounding file-descriptor and mapped-memory usage in long-running processes

Under The Hood

Architecture smmap is organized into a few focused modules. mman.py (~590 lines) implements the memory managers, including a SlidingWindowMapManager that maps fixed-size windows on demand and a StaticWindowMapManager that maps whole files, both tracking a pool of MapRegion objects and reference counts per client. util.py provides the MapRegion, MapWindow, and allocation helpers, buf.py exposes a string-like SmmapBuffer over a managed cursor, and __init__.py re-exports the managers and buffer at the package root. When a client requests a byte range, the manager finds or creates a covering region, bumps its refcount, and evicts least-recently-used regions once limits are exceeded.

Tech Stack Pure Python targeting 3.7+, depending only on the standard library’s mmap module. Packaging uses setuptools via setup.py/setup.cfg, testing runs through tox, and documentation is published on Read the Docs. No third-party runtime dependencies are required.

Code Quality The package includes a dedicated test suite under smmap/test/ covering the manager, buffer, utilities, and a tutorial-style test. Modules are small and single-responsibility, and the code is mature and stable. The README is candid about limitations, notably that file handles can leak due to reliance on __del__ and that access is read-only by design.

API Design Two layers of API are offered: the native interface (managers plus cursors) for performance-critical code, and the higher-level SmmapBuffer that behaves like a read-only string for convenience. Managers expose tunable window sizes and limits, and the simplified static manager trades sliding behavior for whole-file mapping on 64-bit systems. The API is compact and well documented, though it targets users comfortable with memory-mapping concepts.