Deploy Google Cloud log export to Splunk using Terraform

The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.

Terraform code to make the Google Cloud Platform to Datadog log collection integration easier.

This project provides a complete Terraform setup to automate the deployment of a real-time Change Data Capture (CDC) pipeline on Google Cloud. It streams data from a Cloud SQL for MySQL source to BigQuery, using Google Cloud Storage (GCS) as a staging area and a Dataflow streaming job for processing and enrichment.

End-to-End Google Cloud Data Engineering Project using Terraform, BigQuery, Dataproc, Cloud Composer 3, Apache Airflow, and Spark.

✈️ Real-time Flight Data Pipeline: Cloud Engineering project using Terraform, Apache Beam (Dataflow), BigQuery, and Dataform for automated passenger segmentation and data lineage.

Dataflow job subscriber to PubSub subscription. It takes message from subscription and push it into BigQuery table.

Real-time Paris mobility pipeline on GCP. Pub/Sub → Dataflow (Apache Beam) → BigQuery (raw/curated/marts) with DLQ & replay. dbt analytics engineering, Terraform IaC, CI quality gates (ruff/mypy/pytest), Looker Studio. Fully reproducible: make deploy from zero.

Terraform Automation for Dataflow Batch & Streaming

Reference architecture for a compliant enterprise data platform on GCP — BigQuery, Dataflow, Composer, dbt, Data Vault 2.0

Real-time data engineering pipeline on GCP for YouTube comments sentiment analysis with Pub/Sub, Dataflow, BigQuery, Airflow, Terraform, Jenkins, and Python.

Arquitectura GCP ETL + CI/CD con integración AS/400 – por Carlos Crudo

This Terraform module sets up a workflow where files stored in Cloud Storage trigger events in Eventarc, which then processes the files and stores the data in Spanner.