Data Platform Architect | Cloud Data Specialist (AWS) | Builder of Reliable Systems
I design and build modern data warehouses, lakehouse platforms, and real-time event streaming systems that analysts trust and engineers enjoy maintaining. While my core expertise is in AWS data architecture, ETL/ELT automation, and performance tuning, I also build full-stack AI applications and modern web platforms.
Connect:
- 🌐 Portfolio: krishnanandanil.com
- 💼 LinkedIn: linkedin.com/in/krishnanand-anil
- 📧 Email: krishnanandpanil@gmail.com
GitHub natively renders these diagrams. If you are viewing the raw file, switch to preview mode.
Medallion architecture utilizing Apache Iceberg on S3, orchestrated via Airflow and dbt.
flowchart TD
subgraph Sources [Data Sources]
A[PostgreSQL / MySQL]
B[SaaS / REST APIs]
C[Flat Files / Logs]
end
subgraph Lakehouse [Data Lakehouse: AWS S3 + Apache Iceberg]
D[(Bronze Layer: Raw Data)]
E[(Silver Layer: Cleaned & Filtered)]
F[(Gold Layer: Business Aggregates)]
end
subgraph Processing [Processing & Orchestration]
G[Apache Airflow]
H[AWS Glue / PySpark]
I[dbt]
end
subgraph Serving [Serving & Analytics]
J[Amazon Athena]
K[(Amazon Redshift DWH)]
L[BI Dashboards]
end
A & B & C -->|Ingestion| D
G -.->|Orchestrates| H
G -.->|Orchestrates| I
D -->|AWS Glue / Spark| E
E -->|dbt Transformations| F
F -->|Serverless Query| J
F -->|COPY / External Schema| K
J --> L
K --> L
style Sources fill:#f9f9f9,stroke:#333,stroke-width:2px
style Lakehouse fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
style Processing fill:#fff2e6,stroke:#ff9900,stroke-width:2px
style Serving fill:#e6ffe6,stroke:#33cc33,stroke-width:2px
Event-driven architecture decoupling source databases from downstream analytics with sub-second latency.
graph LR
subgraph "Transactional Systems"
DB[(Amazon Aurora / RDS)]
end
subgraph "Streaming & Compute Infrastructure"
CDC[Debezium / AWS DMS]
Kafka[Apache Kafka / Kinesis]
StreamProc[Spark Streaming / Lambda]
end
subgraph "Downstream Consumers"
RT_DB[(DynamoDB<br/>Fast Lookups)]
DWH[(Redshift<br/>Micro-batch)]
end
DB -->|Change Data Capture| CDC
CDC -->|Publish Events| Kafka
Kafka -->|Subscribe| StreamProc
StreamProc -->|Sub-second Latency| RT_DB
StreamProc -->|5-min Refresh Cycle| DWH
classDef streaming fill:#0052CC,stroke:#FFFFFF,stroke-width:2px,color:white;
class CDC,Kafka,StreamProc streaming;
Bridging enterprise data with Large Language Models for Natural Language Querying (NLQ).
graph TD
subgraph "Enterprise Data Foundations"
DWH[(Redshift DWH)]
Docs[Internal Docs / Confluence]
end
subgraph "Processing Pipeline"
Chunk[Chunking & Processing]
Emb[Embedding Model]
end
subgraph "AI / GenAI Infrastructure"
VecDB[(Vector Database)]
LLM[LLM / Foundation Model]
end
subgraph "User Interface"
Chat[Self-Service NLQ UI]
end
DWH & Docs --> Chunk
Chunk --> Emb
Emb -->|Store Embeddings| VecDB
Chat -->|1. User Question| LLM
LLM -->|2. Semantic Search| VecDB
VecDB -->|3. Context Retrieval| LLM
LLM -->|4. Synthesized Answer| Chat
classDef ai fill:#6B4E71,stroke:#FFFFFF,stroke-width:2px,color:white;
class Emb,VecDB,LLM ai;
- ResumeForge-AI
An AI-powered resume generation tool that turns standard bullet points into FAANG-worthy achievements. Demonstrates practical integration of Generative AI, LLMs, and prompt engineering in a functional application.
- portfolio_sveltekit
My personal portfolio and blog architecture. A modern, highly performant web application built with SvelteKit and deployed on Cloudflare Pages utilizing Server-Side Rendering (SSR). - portfolio-angular
An alternative frontend architecture implementation utilizing Angular, demonstrating component-based UI design.
- Abalone_classification_regression
End-to-end Exploratory Data Analysis (EDA), regression, and classification models applied to the Abalone dataset using Python. - Flower-recognition-Keras_sequential
A deep learning computer vision model built using Keras Sequential API to accurately classify flower species.
(Note: My large-scale enterprise data engineering architectures are proprietary and closed-source, but you can read detailed architectural breakdowns on my Portfolio.)
Cloud & Infrastructure (AWS): S3, Athena, Glue, EMR, Lambda, Kinesis, Redshift, Aurora PostgreSQL, DynamoDB, IAM, Terraform, Docker, Kubernetes (K8s)
Data Engineering: Apache Kafka, Debezium (CDC), Apache Airflow, dbt, Spark/PySpark, Hadoop, ETL/ELT
Architecture Patterns: Event-Driven Architecture, Microservices, Medallion Data Lakes, Dimensional Modeling, Reference Architectures
App & Web Dev: Python, SQL, TypeScript, SvelteKit, Angular, Flutter, REST/GraphQL APIs
AI/ML: RAG, Vector Databases, Keras, Pandas, Scikit-learn
- Metadata-driven warehouse automation: Treating data ownership, tests, and lineage as code.
- Agentic AI Architecture: Using specialized LLM agents for data quality anomaly detection and automated documentation.
- Advanced Lakehouse Patterns: Schema evolution and time travel with Apache Iceberg on S3.
- “SELECT *” is fine—as long as you know why you’re doing it.
- A well-modeled schema will always beat a fancy dashboard.
- The best data pipelines are the ones you forget exist because they never break.
"Good data models are like good jokes — if you have to explain them, they’re not working."
If you see something interesting in my repos, clone it, break it, and make it better.