Simple stream processing pipeline
-
Updated
Jun 17, 2024 - Python
#
datapipeline
Here are 93 public repositories matching this topic...
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
python data etl openai datapipeline etl-framework unstructured pinecone cohere etl-pipeline weaviate vector-database qdrant qdrant-vector-database
-
Updated
Oct 5, 2024 - Python
High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.
-
Updated
Apr 22, 2022 - Python
kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
-
Updated
Jan 6, 2023 - Python
Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like
-
Updated
Apr 23, 2023 - Python
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.
database etl extractor google-sheets data-engineering data-lake business-intelligence dwh elt datapipeline data-pipeline extract-transform-load usql excel-to-sql etl-job etl-process sharepoint-integration sharepoint-sync sync-google-sheet
-
Updated
May 29, 2025 - Python
Материалы для курса Введение в Data Engineering: дата пайплайны
-
Updated
Feb 18, 2024 - Python
-
Updated
Aug 5, 2021 - Python
Simple Airflow on Kubernetes (GKE)
docker kubernetes airflow gcp google-cloud datascience gke datapipeline kubernetes-executor airflow-kubernetes
-
Updated
Sep 17, 2021 - Python
DBT and clickhouse test project with dagster
-
Updated
Aug 29, 2023 - Python
This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
-
Updated
Jan 26, 2021 - Python
An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai
-
Updated
Nov 21, 2023 - Python
This is a project which demonstrates creation of a data pipeline by scraping data using twitter API and creating a data delivery stream using Kinesis Firehose for ingesting data to Amazon S3.
python aws data big-data etl twitter-api aws-s3 bigdata kinesis python3 kinesis-firehose boto3 datapipeline real-time-processing aws-glue
-
Updated
May 4, 2020 - Python
Dagster Tutorial to get you started with Dagster as an absolute Beginner. The tutorial covers various topics like Dagster Installation, Dagster Asset, Dagster Job, Dagster Scheduler, Dagster Ops, and more. It is completely free on YouTube and is beginner-friendly without any prerequisites.
-
Updated
Nov 26, 2023 - Python
python aws lambda aws-lambda dynamodb s3 s3-bucket stream-processing datapipeline iam-policy data-engineer iam-role iam-credentials aws-event
-
Updated
May 28, 2022 - Python
An Automated data pipeline using "Apache Airflow" performing "ETL" on RAW data using "Pandas" library then stage data into "PostgreSQL" then process it distributed cluster and parallelly using "Spark" and loaded final useful data into "ElasticSearch" NoSQL DB warehouse
python elasticsearch kibana sql spark apache-spark etl nosql postgresql pandas pyspark data-engineering dag datapipeline apache-airflow
-
Updated
Jan 2, 2022 - Python
A data pipeline to daily pull public transport data from the opentransportdata.swiss portal. This pipeline has three tasks, pull the right data from opentransportdata.swiss, push the data to s3 for storage, and transform and load the transformed data to a database. Hopefully this repository helps people explain ETL / Batch data pipeline.
-
Updated
Mar 28, 2018 - Python
Improve this page
Add a description, image, and links to the datapipeline topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the datapipeline topic, visit your repo's landing page and select "manage topics."