Open Source Data Engineering Apps
Discover open source data engineering tools for building reliable data pipelines, ETL processes & scalable data storage. Unlock the power of your data!
Apps in Data Engineering
Apache Airflow
Data Engineering
Define, schedule, and monitor complex data workflows as Python code — with a powerful UI, 80+ provider integrations, and battle-tested scalability across thousands of production deployments.
Airbyte
Data Engineering · Developer Tools
Open-source ELT platform with 600+ connectors for moving data from any source to warehouses, lakes, and AI agents.
ClickHouse
Analytics · Data Engineering · Databases
Open-source column-oriented database that delivers real-time analytical queries on petabyte-scale data with millisecond latency.
Redash
Analytics · Data Engineering
Redash lets anyone connect to 35+ SQL and NoSQL data sources, write a query in the browser, and turn the result into a shared dashboard — no separate BI suite required.
WrenAI
AI Agents · Analytics · Data Engineering
Open-source GenBI engine that lets AI agents turn natural-language questions into governed SQL, charts, and shareable dashboards across 20+ data sources — no vendor lock-in, no black-box prompts.
marimo
Data Engineering · Developer Tools
A reactive Python notebook that eliminates hidden state, runs reproducibly, and deploys as a web app or script — stored as pure Python, built for the AI era.
Timeplus Proton
Analytics · Data Engineering
Single C++ binary SQL engine for real-time stream processing, ETL, and analytics on Kafka, Redpanda, and ClickHouse with sub-millisecond latency.
sirchmunk
AI Development · Data Engineering
Drop your files and search them instantly — no vector DB, no indexing pipeline, just raw data queried by a self-evolving intelligence layer.
Flowfile
Data Engineering
Visual ETL that compiles to Polars — build pipelines on a canvas, export as standalone Python, and run anywhere without platform lock-in.
Docglow
Data Engineering
A next-generation documentation site generator for dbt Core projects — lineage explorer, health scoring, and full-text search for teams without access to dbt Cloud's built-in docs features.
argilla
AI Development · Data Engineering
Collaborate on high-quality AI training data with a self-hosted annotation platform built for LLMs, NLP, and multimodal models.
SWIRL
Data Engineering · Databases · Search
Federated AI search and RAG across 100+ enterprise sources—no data extraction, no vector database required.
openduck
Data Engineering · Databases
OpenDuck brings MotherDuck-style cloud capabilities to self-hosted DuckDB — attach remote databases, run hybrid queries across local and remote nodes, and own your data with an open gRPC and Arrow IPC protocol.
About Data Engineering
Data engineering focuses on building and maintaining robust data pipelines that enable organizations to make data-driven decisions. These tools are essential for turning raw data into actionable insights, automating data workflows, and ensuring data quality.
Typical features within this category include:
- Data Integration: Connecting to various data sources (databases, APIs, cloud storage) and ingesting data.
- Data Transformation: Cleaning, validating, enriching, and transforming data into usable formats using techniques like ETL (Extract, Transform, Load).
- Data Storage: Managing and organizing data in efficient and scalable storage systems (data warehouses, data lakes).
- Data Pipeline Automation: Scheduling and monitoring data workflows to ensure reliability and consistency.
- Data Quality & Governance: Implementing checks and balances to maintain data accuracy, completeness, and security.
Data engineering solves critical problems such as siloed data, inefficient workflows, and a lack of reliable data for analytics. By streamlining the data process, organizations can unlock business value faster, improve decision-making accuracy, and gain a competitive edge. Furthermore, robust data pipelines are foundational for machine learning initiatives, enabling teams to build and deploy predictive models with confidence.
