promise-pool

@supercharge/promise-pool is a small TypeScript library for running async operations over an array or iterable with a controlled concurrency limit, instead of firing every promise at once with Promise.all or serializing them one by one in a for loop. It exposes a fluent, chainable API modeled after Array.prototype.map: call .for(items), set .withConcurrency(n), and pass an async callback to .process(), and the pool schedules a fixed number of tasks in flight at any time.

Beyond basic throttling, the pool supports custom error handling via .handleError() so failures don’t abort the whole batch, per-task timeouts via .withTaskTimeout(), progress callbacks via .onTaskStarted()/.onTaskFinished(), the ability to stop processing early from within a task, and an opt-in mode that keeps each result aligned with its source item’s index. It ships with full TypeScript type definitions and no runtime dependencies.

What You Get

• Fluent PromisePool builder API via .for(items).withConcurrency(n).process(fn)
• Custom error handling via .handleError() that runs without aborting the whole batch
• Per-task timeouts via .withTaskTimeout(ms), independent of overall pool concurrency
• Progress hooks via .onTaskStarted()/.onTaskFinished() plus processedCount()/processedPercentage() stats
• Corresponding results mode that preserves source-item ordering with notRun/failed symbols
• Full TypeScript type definitions and zero runtime dependencies

Common Use Cases

• Batching API calls to a rate-limited third-party service without exceeding N in-flight requests
• Processing a large dataset (e.g. rows from a CSV or DB cursor) with bounded memory and concurrency
• Fanning out multiple async writes, like file uploads, with progress reporting
• Retrying or classifying failures per item without stopping the whole batch

Under The Hood

Architecture The public PromisePool class (src/promise-pool.ts) is a thin, chainable builder that stores configuration (concurrency, timeout, error handler, progress handlers) and, on .process(), hands everything off to the internal PromisePoolExecutor (src/promise-pool-executor.ts), which holds a single mutable meta object (items, concurrency, results, errors, active tasks array) and drives an async iteration loop: process() pulls items one at a time, calls startProcessing() to kick off a task, and awaits waitForProcessingSlot() (a Promise.race over active tasks) before pulling the next item once the concurrency limit is reached. Shared interfaces in src/contracts.ts (Stoppable, UsesConcurrency, Statistics) are implemented by the executor and exposed to user callbacks, so a task handler can call pool.stop() or read pool.activeTasksCount() without seeing internal state. Error handling funnels through handleErrorFor(), which distinguishes an internal StopThePromisePoolError control-flow signal from a fatal ValidationError from a normal processing error routed to the user’s .handleError() or the default error-collection array — changing this core task lifecycle would ripple into the corresponding-results indexing, timeout racing, and progress-handler firing that all hook off the same startProcessing/waitForProcessingSlot path.

Tech Stack Written in TypeScript and compiled with tsc to a dist/ folder that’s the sole published files entry (main/types both point to dist), with sideEffects: false for tree-shaking. The package has zero runtime dependencies — its devDependencies are limited to TypeScript, @supercharge/eslint-config-typescript plus eslint for linting, and uvu with expect for testing and c8 for coverage. CI runs via GitHub Actions across a Node.js 20/22/24 matrix on Ubuntu, executing npm run test:full (build, lint, then coverage-instrumented tests) on every push and pull request.

Code Quality The test suite is extensive, spanning three files that exercise core pool behavior, iterable/async-iterable support, and the stop-the-pool control flow, including edge cases like validation errors, custom error-handler semantics, task timeouts, and corresponding-results mode. Tests run through the lightweight uvu runner paired with the expect assertion library, and are linted as part of CI via the shared Supercharge ESLint config. Errors are modeled with dedicated typed classes (ValidationError, PromisePoolError, StopThePromisePoolError) rather than generic throws, giving callers a way to discriminate error causes. Method and file naming is consistent (kebab-case files, camelCase methods), and nearly every public and private method carries a JSDoc comment describing its behavior and intent.

What Makes It Unique The standout feature is .useCorrespondingResults(), which keeps each result at the same array index as its source item even though tasks complete out of order under concurrency, using dedicated PromisePool.notRun and PromisePool.failed symbols to mark items that never ran or threw — a detail that plainer concurrency-limiter utilities typically leave to the caller to reconstruct. It’s paired with the ability to call pool.stop() from inside either the processing callback or the error handler to halt further intake while still returning whatever has already been computed, and a per-item timeout that races independently of the pool’s overall concurrency setting. These are thoughtful, ergonomic refinements on the well-established bounded-concurrent-map pattern rather than a new abstraction.