etl-framework

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Logstash - transport and process your logs, events, or other data

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Flow-based programming for JavaScript

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

This repository is a getting started guide to Singer.

Making data lake work for time series

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

A simplified, lightweight ETL Framework based on Apache Spark

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Knowledge Graph Toolkit

A Task-Based Workflow Orchestration Engine

A tool for building feature stores.

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

Structured Data Extractor for AI Agents. Search your documents or the web for specific data and get it back in JSON or Markdown in a single tool call.

Bender - Serverless ETL Framework

Focusing on building industry-leading ETL engines.