All 131 Dependencies
Every package PeerDB depends on, ranked by repo health score.
PeerDB is an open-source ETL/ELT tool purpose-built for streaming data out of PostgreSQL rather than treating it as one generic connector among hundreds. Instead of a YAML config file or a SaaS console, PeerDB fronts its pipelines with `nexus`, a Rust-built server that speaks the Postgres wire protocol on port 9900 — so a pipeline (a "mirror") is created, monitored, and torn down with plain SQL statements like `CREATE MIRROR`, run from psql, pgAdmin, a BI tool, or a migration framework like Flyway.
Under the hood, a Go service (`flow`) defines Temporal workflows — `CDCFlowWorkflow`, `QRepFlowWorkflow`, `SnapshotFlowWorkflow`, `SetupFlowWorkflow`, `DropFlowWorkflow`, and `MaintenanceFlowWorkflow` — that orchestrate the actual data movement. Sources are captured via PostgreSQL logical replication (WAL), MySQL binlog (GTID or file-position), or MongoDB change streams; a dedicated snapshot worker parallelizes the initial full-table backfill by partition before CDC takes over for ongoing changes. Destinations include Snowflake, BigQuery, ClickHouse, PostgreSQL, S3, Kafka, Event Hubs, Pub/Sub, and Elasticsearch, with a catalog Postgres database tracking peers, flows, table-schema mappings, and per-batch state, and MinIO staging AVRO files between pull and load.
The project puts real engineering into problems that generic ETL tools often gloss over for Postgres specifically: efficient streaming of large TOASTed columns without re-fetching whole rows, tuned logical-replication configs and parallel WAL-slot reads for 10x faster CDC, and automatic detection and replay of source schema changes (tracked in a `schema_delta_audit` table). Fault tolerance comes from Temporal's built-in state persistence, retries, and crash recovery, with Slack and email alerting wired in for mirror failures.
PeerDB is licensed AGPL-3.0 and the self-hosted stack (Docker Compose or Tilt for local dev) ships with the full connector set and no license-gated features; PeerDB Cloud (peerdb.io) is a separately hosted managed offering built on the same open-source core, and the project is also available natively inside ClickHouse Cloud's ClickPipes.
