Navya Vats

Everyone talks about prompting and embeddings when they talk about AI engineering. Almost nobody talks about the boring skill that actually determines whether your RAG system survives production: pipeline discipline. Six years in data engineering taught me things I didn't expect to reuse — until I started building RAG systems: 🎯 Data quality obsession. Bad chunks in, bad answers out. The same rigor I apply to source data validation in ETL pipelines is exactly what separates a demo RAG from a reliable one. 🔄 Idempotency and reprocessing. Embeddings drift, documents get updated, sources change. If your ingestion pipeline can't safely re-run without duplicating or corrupting the vector store, it isn't production-ready — full stop. 📊 Monitoring what actually matters. I built an ETL monitoring system that cut audit costs 99% by watching the right signals, not everything. RAG systems need the same discipline — retrieval quality, latency, hallucination rate — not just "is the API up." 🏗️ System design over model design. Most AI engineering failures I've seen aren't model failures. They're pipeline failures: bad orchestration, no retry logic, silent data drift. If you're a data engineer eyeing a move into AI engineering: you're closer than you think. The hard part was never the LLM — it was always the data. #AIEngineering #DataEngineering #RAG #CareerTransition #MachineLearning

10

6 Comments

Ladders of Causation | A Growth Product Data Science Lens Just finished 'The Book of Why' by Judea Pearl, and it gave me a structured framework to think about applications of causal inference to product data science and growth. Pearl's Ladder of Causation has three rungs: 🔹 Seeing | Observation: 1) Users who engage with Feature X retain better 2) User segments offered promotions convert better Both patterns are real. Neither tells us whether they caused the outcome! Even if dashboards are green and all is well, is the data enough to act without more evidence? What is the opportunity cost of the decision to be made? In most cases, observational data is powerful and can help: • prioritize opportunities • size bets • decide what's worth intervening on 🔹 Intervening | A/B Testing: We run a test. The group that received Feature X retains better, or the discount group converts at a higher rate. Now we answer harder questions, especially in high-leverage decisions: • Which users actually responded? • Which discount depth drove incremental revenue? • Did new and returning users respond differently? Experiments are powerful but not free: bandwidth, engineering cost, traffic allocation, opportunity cost. In principle, I recommend running tests when decision leverage justifies the cost. Impact estimates built on well-designed interventions are unbiased and bring you closer to knowing what to do next in the highly iterative process of product growth. 🔹 Imagining | Counterfactual Reasoning: Pearl said, “Counterfactual learners can imagine worlds that do not exist.” This is the forward-looking rung. As an example, a structural causal model (SCM) starts with explicit assumptions about how variables relate: • How does personalization interact with pricing? • How does discount depth interact with content quality? • How driver incentives interact with marketplace supply? A/B experiments (when feasible) can complement and populate these relationships with real estimates, and the output is a simulation you run before committing to a decision. Of course, the assumptions are as important as the data! Each of these rungs has a place in driving growth and product decisions. I recommend flexing between these three rungs based on factors such as: • decision reversibility • cost of error • urgency • feasibility of experimentation All opinions are my own and intended for data science community engagement and learning.

12

What things in Gen AI a Data Engineer should know! LLM - Large Language Model (act's like a brain) This LLM has 2 main stages - Pre training - Fine tuning although there are more.. In Pre training Big Companies will take a big chunk of data (100's of TB) available on internet, blogs, wikipedia etc and get a gist of it in a few GB's. Lets consider 70 Billion parameter model, each parameter here will be a float of 2 bytes and will take 140 GB. This will take 10's of days and 1000's of heavy machines, and 10's or 100's of million dollars to do this and we get a base/foundation model. This base model cannot answer question, rather if you give it a question it might give more questions. Now, we can leverage the foundation model and develop something. As a Data Engineer, => Prompt Engineering is the easiest thing to do, but when creating applications you need to write it in a programatic way. writing better prompts with more context can help us get better results. => Retrieval Augmented Generation (RAG) - As a Data Engineer you can index your company documents, here you would require to clean the data and index it in a vector database. Later when a prompt is given, it will try to do a simarity search and will search in your vector database. whatever results come will be augmented to your prompt and it's like more context is passed as part of the prompt for better results. So, setting up a vector DB, cleaning documents, and storing in vector DB can be directly in a Data Engineers portfolio. => Model Fine Tuning - This is where the world is moving towards. Instead of striving for bigger and bigger base model, it has been seen that a decent size base model with a good fine tuning done for specific tasks can perform better. So, we will take a foundation mode, and we might have 10's or 100's of thousands of quality questions and answers (high quality training data) on which we fine tune the model. There can be different fine tuned models specific to each task. Do not try using a single LLM for everything. A better way is to use specialised LLM's fine tuned differently to do different stuff's. A model might be good at summarisation, other might be good at judging which summaries are better. Also the LLM's can be given access to various tools like google search, libraries to plot a graph, calculator etc in order to achieve a task. => Now when you develop complex application, it might happen you use a LLM for summarisation, and the output of this LLM goes to other LLM for judging. So there can be a big pipeline, this can be developed using langchain. So I feel - A basic understanding of terminologies around Pre training - Vector Database - RAG - Model Fine tuning to some extent - Lang Chain - Agentic are the things which at the minimum a Data Engineer should know. But there are many more things within this, which I will cover in depth in my "Gen AI for Data Engineers Program" I hope you learnt something new!

89

Applied Statistics - #4 For anyone trying to go 0-> Mastery in next 10 days! Why averaging country-level statistics gives you the wrong global answer Most data scientists get this interview question wrong: "If you know each country's mean height and population, how do you estimate global mean height?" The trap: just averaging the country means. The problem with naive averaging: 1) Treats Luxembourg (640K people) the same as India (1.4B people) 2) Ignores population weights entirely 3) Produces systematically biased estimates The correct approach: Use population-weighted statistics: 1) Global mean = weighted average of country means (weight = population proportion) 2) Global variance = combine within-country variance + between-country variance using proper hierarchical formulas 3) Standard deviation ≠ average of standard deviations (this never works!) This same principle applies to: 1) A/B test analysis across user segments 2) Distributed model training with heterogeneous data 3) Multi-region metric aggregation 3) Any hierarchical/grouped data structure Full breakdown with Python implementation and the exact formulas → 1) https://lnkd.in/gETZP7Ey 2) https://lnkd.in/gRu4tcAy What's a statistics concept you initially got wrong in interviews?

4

1 Comment

Apparently, when you google RBF, the top result is Resting Bitch Face 💀 But in data science land — and especially for those of us working with Bayesian or kernel-based models — RBF stands for Radial Basis Function. And it’s way more charming than its acronym suggests. At its core, an RBF measures similarity between two points: if two points are close, it tends to 1, if they’re far apart, it drops toward 0. Classic example — Gaussian. That distance-based smoothness is what makes RBFs so useful for capturing subtle, non-linear patterns, you can even stack them. RBFs actually power a lot of what we do: 🔔Gaussian Processes and Bayesian modeling (check out PyMC for a great implementation) ➗SVMs where they’re the classic kernel (remember that kernel trick when you “lift” data into a high-dimensional space and get the dot product there for free?) 🕸️Even neural networks, where RBFs can act as hidden activations So next time your data refuses to behave linearly — give it an RBF. This little appreciation post is inspired by Wesley Boelrijk's Bayesian modeling talk at PyData Amsterdam. #DataScience #BayesianModeling #MachineLearning #PyMC #Halloween

267

12 Comments

If there was one clear message from Connected Data London, it’s this: The era of throwing out-of-the-box LLMs and Agents at a problem and hoping for the best is ending. We are seeing a pattern where these rapid POCs are short-lived. They look great in a demo, but they hardly ever make it into production. Why? Because without semantics, they lack the reliability and context required for enterprise use. At the conference, companies from different sector presented their solutions in production and the common differentiator for success is the adoption of contextually relevant data, often modelled as knowledge graphs. Similar debates happened at ISWC - International Semantic Web Conference, and the once forgotten disciplines of Knowledge Engineering and Knowledge Representation are back. They are no longer academic concepts; they are the practical for improving governance, reliability and trustworthy of AI responses, and improving explainability and safety on automated decision making. Glad to connect with many colleagues: Audrey Depeige Anahita Pakiman Charles Ivie Ora Lassila Vivek Jain Ian Robinson Nicole Moldovan #CDL2025 #KnowledgeEngineering #AI #ISWC

94

If you’re thinking about moving into AI roles, this is worth your time — Divij Bajaj is sharing from first-hand experience, not theory.

14

Quantization in Large Language Models (LLMs) As LLMs scale, intelligence is no longer the bottleneck — memory, latency, and cost are. Quantization turns out to be one of the most powerful (and misunderstood) techniques enabling real-world deployment of large models. In this blog, I break down: • Why LLMs are surprisingly quantization-friendly • INT8 vs INT4 vs NF4 — what actually works in practice • How modern architectures absorb low-bit noise • Where quantization fails and how to design around it Quantization is no longer an optimization step — it’s foundational infrastructure for production LLM systems. Link: https://lnkd.in/gqGje75G #LLMs #AIEngineering #Quantization #MachineLearning #GenAI #MLOps

9

2 Comments