This project implements a real-time healthcare data lakehouse system to predict whether a patient is ill or not based on simulated medical test data. It uses a combination of synthetic training data (static) and real-time simulated input to apply machine learning in a streaming context.

• Synthea generates static patient records for model training.
• Faker simulates real-time medical test results to mimic streaming patient records.
• Apache Kafka handles the real-time ingestion of records.
• Apache Spark Structured Streaming processes streaming data and applies a trained ML model to classify each record as "ill" or "healthy".
• Delta Lake stores the processed data in a structured multi-layered format (Bronze → Silver → Gold).
• Google BigQuery supports querying and visualization.
• Looker Studio is used for analytics dashboards and reporting.

• Generate realistic training data using Synthea.
• Simulate real-time patient test results using Faker and ingest via Kafka.
• Build a real-time illness prediction pipeline using Spark Streaming and a pre-trained model.
• Store data in Delta Lake with a multi-stage architecture.
• Push final results to BigQuery for analysis and reporting.

• Source: synthea_data/training_data.csv
• Description: High-quality synthetic patient data generated by Synthea, representing realistic vitals and clinical conditions.
• Usage: Machine learning model training (RandomForestClassifier)

Key Features Used in Training:

• Source: src/streaming_data_producer.py
• Description: Simulates real-time patient medical test results and publishes them as JSON records to a Kafka topic (patient_data), emitting one record per second.
• Purpose:
  • Emulates a live feed of new patient records arriving in real time
  • Feeds Spark Structured Streaming for ingestion, transformation, and ML-based illness prediction

Fields in Each Streamed JSON Record:

• Table: patient_data.gold_patient_data
• Mirror of the Delta Lake Gold table
• Used for analytics and dashboards

• Patient Age Overview
• Lab Tests by Age Group
• Gender-Based Differences

Predict illness status of a patient based on medical test results.

1. Preprocess data using prepare_training_data.py
2. Handle class imbalance using SMOTE
3. Train a Random Forest classifier (train_model.py)
4. Save the trained model using joblib

1. synthea_data/ – Static Synthea-generated patient data used for model training (training_data.csv).
2. src/prepare_training_data.py – Cleans and transforms static data for training.
3. src/train_model.py – Trains the classification model and saves it.
4. models/illness_predictor.pkl – Serialized trained model for real-time inference.
5. src/streaming_data_producer.py – Simulates real-time patient test records using Faker.
6. delta_lake_setup/schema_bronze.json – Schema for Bronze (raw) layer.
7. delta_lake_setup/schema_silver.json – Schema for Silver (cleaned) layer.
8. delta_lake_setup/schema_gold.json – Schema for Gold (classified) layer.
9. src/streaming_inference_job.py – Spark job that ingests from Kafka, writes to Bronze → Silver → Gold layers, and loads curated Gold data into BigQuery for reporting.
10. notebooks/project_walkthrough.ipynb – Interactive setup guide and walkthrough of key components.
11. notebooks/reporting.md – Visualization and reporting examples.

• This project is for educational purposes only.
• All data is synthetically generated using Synthea and Faker.
• It is not intended for clinical or medical use in any real-world setting.

• Synthetic Data Only: All input is generated with Synthea and Faker, which may lack the complexity, variability, and edge cases of real clinical data.
• Offline Model Retraining: The ML model is trained offline and used as-is during streaming. There is no online learning or adaptive update mechanism.
• Basic Feature Set: The current feature set doesn’t account for patient history, comorbidities, or longitudinal trends.
• No Model Monitoring: There is no performance monitoring or drift detection in the live prediction phase.
• Latency Not Optimized: The streaming job is batch-based and may not achieve true low-latency performance in all configurations.

• Incorporate Longitudinal Data: Enhance predictions using patient history and temporal trends from repeated visits.
• Introduce Model Monitoring: Integrate tools like MLflow, Prometheus, or custom drift detection to monitor prediction quality over time.
• Online or Incremental Learning: Explore real-time model updates using streaming-compatible frameworks like River.
• Real-Time Alerting: Push prediction results to alerting platforms (e.g., Slack, PagerDuty) for operational simulation.
• Enhanced Visualization: Create interactive dashboards in Looker Studio for patient-level drilldowns, anomaly detection, and model performance summaries.
• Governance & Metadata Tracking: Add Apache Atlas or Data Catalog for lineage, schema versioning, and audit readiness.

This project showcases a real-time healthcare data lakehouse architecture for illness prediction using a modern big data stack. By combining Synthea-generated training data with real-time simulated patient vitals, it demonstrates how machine learning can be applied to streaming medical data in a scalable and modular fashion.

Key highlights include:

• Real-time ingestion and processing using Kafka and Spark Structured Streaming.
• Delta Lake multi-layered architecture (Bronze, Silver, Gold) for efficient and reliable data storage.
• Integration with BigQuery and Looker Studio for real-time analytics and visualizations.
• Application of a Random Forest classifier to predict patient health status based on clinical vitals.

This end-to-end pipeline represents a foundational blueprint for future real-world healthcare streaming systems — balancing engineering, machine learning, and analytics in one solution.