A GitHub repo that contains everything related to MLOps.
• Week 0: Project Setup
• Week 1: Model Monitoring - Weights and Biases
• Week 2: Configurations - Hydra
• Week 3: Data Version Control - DVC
• Week 4: Model Packaging - ONNX
• Week 5: Model Packaging - Docker
• Week 6: CI/CD - GitHub Actions
• Week 7: Container Registry - AWS ECR
• Week 8: Serverless Deployment - AWS Lambda
• Week 9: Prediction Monitoring - Kibana
If you're serious about deploying ML models in the real world, this is a great place to start.
#MLOpshttps://lnkd.in/gtzth7uX
🚀 𝐇𝐨𝐰 𝐖𝐞 𝐌𝐚𝐝𝐞 𝐒3 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 80% 𝐅𝐚𝐬𝐭𝐞𝐫 (𝐚𝐧𝐝 𝐂𝐡𝐞𝐚𝐩𝐞𝐫!)
When our boto3 scripts started slowing down under millions of S3 files, we knew it was time to rethink ingestion from the ground up.
💡 Working with Rohan Giriraj, we rebuilt our entire approach using S3 Batch Copy — shifting ingestion from compute-heavy scripts to AWS’s managed orchestration.
The result:
⚙️ 100K+ files/min throughput
💰 ~60% lower cost (no EC2 retries!)
🌍 Cross-region copies made native
📈 Zero manual retries — fully managed, fully scalable
We learned that boto3 gives flexibility, but Batch Copy gives velocity — and the peace of mind that comes with it.
👉 Read full story on Medium: https://lnkd.in/gq7aCQg6#DataEngineering#AWS#S3#BatchCopy#DataPlatform#AutodeskTechTrailblazers#Ingestion#BigData#Automation
The free MLOps with Databricks course skips the theory and dives straight into production.
10 short, hands-on lectures covering:
▫️MLOps fundamentals
▫️Databricks workflows
▫️MLflow tracking & model registry
▫️Serving architectures
▫️Endpoint deployment
▫️CI/CD & monitoring
Built on Databricks + MLflow - the same tools teams use in real production pipelines.
https://lnkd.in/gkGZw3NR
Secure MLOps platforms are a must for efficient ML use case productionization.
• Terraform and GitHub enable a secure, multi-account setup with strict security constraints and automatic deployment using CI/CD technologies.
• Custom Amazon SageMaker Projects templates provide example repositories for deploying ML services using Terraform.
• End-users can select and customize templates to fit their use case, streamlining the journey from model development to deployment.
Takeaway: Implementing a secure MLOps platform with Terraform and GitHub enables reproducibility, robustness, and end-to-end observability of ML use cases.
#mlops#terraform#github#machinelearning
🚀 Levelling Up with GitHub Workflows in Databricks
If you're automating your CI/CD with GitHub, mastering Workflows is essential. Here’s a quick breakdown of features that have helped streamline and scale our pipelines:
✅ Workflows
YAML-based automation triggered by events like push, PR, or schedules. Handles everything from tests to deploys.
🔁 Reusable Workflows
Write once, use everywhere. Ideal for standardizing steps across projects:
🧩 Custom Actions
Create your own logic using JavaScript or Docker. Great for secrets, tools, or integrations tailored to your stack.
💡 Variables, Contexts & Expressions
Dynamic logic using env, secrets, github contexts
Expressions with conditional statements
⚡ Concurrency & Caching
Avoid duplicate runs using concurrency
Speed up builds with actions/cache for deps (npm, pip, etc.)
📦 Artifacts
Store or pass files between jobs (e.g. test reports, coverage):
🚀 Deployment Environments
Use environments for gated releases, scoped secrets, and approvals for staging, prod, etc.
🔗 Databricks Asset Bundles (DAB) Integration
You can deploy Databricks Asset Bundles directly from GitHub Actions using the Databricks CLI (databricks bundle deploy) in your workflow. Combine this with environment-specific secrets and review gates to enable smooth, automated deployments to dev, staging, and prod workspaces. Clean, reproducible, and version-controlled.
✅ Official Github Actions Documentation
https://lnkd.in/gxqBRRiR
✅ For detailed info on DAB integration with github actions check the link
https://lnkd.in/gGNcuavd
✅ GitHub Actions has grown into a powerful CI/CD platform. Use these tools to build faster, deploy safer, and scale with confidence.
#GitHubActions#CI#CD#DevOps#Automation#Databricks#DataEngineering#DataEngineer
🔥 Meet Confluent for VS Code 2.0
We're proud to announce the newest release of Confluent for VS Code, a major milestone that streamlines the development of Flink UDFs inside VS Code:
➡️ Full Flink UDF lifecycle support: scaffold, develop, test, deploy, register, and use UDFs, all from within VS Code.
➡️ Built-in project templates to jumpstart Flink UDF development.
➡️ Automatic UDF discovery: the extension now parses .jar artifacts and auto-populates available class names for registration.
➡️ Rich metadata for registered UDFs, see parameters and signatures at a glance just by hovering on the UDF.
➡️ Enhanced Flink SQL authoring: IntelliSense support for referencing UDFs, plus Copilot code completion.
Check out the extension's source code on GitHub (https://lnkd.in/ebnYmPuV) or install it from the marketplace (https://lnkd.in/euv2fPsC).
6 reasons ML engineers should use GitHub Actions
(90% miss number 5)
1– Automate your workflows
→ Run tests, lint code, and build containers automatically whenever you push to GitHub.
2– Keep models reproducible
→ Ensure each commit trains, evaluates, and logs results in the same controlled environment.
3– Simplify CI/CD
→ Build, test, and deploy models or APIs straight from GitHub, no manual steps needed.
4– Integrate with cloud & containers
→ Trigger deployments to AWS, GCP, or Azure, or push Docker images to registries without leaving your repo.
5– Catch issues early
→ Run pytest, data validation, and static checks before merging so broken pipelines never hit main.
6– Collaborate with confidence
→ Review every change with automated checks, artifacts, and logs attached to each pull request.
A few things to know about GitHub Actions:
✅ Hosted on GitHub
✅ Configured with YAML
✅ Easy to set up & maintain
✅ Public repos → Free (with limits)
✅ Private repos → Some free mins, then paid
✅ Limited by GH infra (or use self-hosted runners)
—
GitHub Actions turns your ML repo into a self-running machine.
Train, test, and deploy seamlessly with every commit.
📌 Save this for your next project.
♻️ Share this to help your network in data.
𝐖𝐞 𝐫𝐞𝐩𝐥𝐚𝐜𝐞𝐝 𝟖 𝐭𝐨𝐨𝐥𝐬 𝐰𝐢𝐭𝐡 𝟏 𝐝𝐚𝐬𝐡𝐛𝐨𝐚𝐫𝐝
Day 9 of #Kuberns100Days
Most dev teams juggle:
❌ GitHub Actions (CI/CD)
❌ AWS Console (Infrastructure)
❌ DataDog (Monitoring)
❌ Sentry (Error tracking)
❌ CloudWatch (Logs)
❌ Terraform (IaC)
❌ Docker Hub (Container registry)
❌ PagerDuty (Alerts)
That's 8 logins, 8 dashboards, 8 billing systems.
And they barely talk to each other.
With Kuberns?
✅ Deploy from GitHub
✅ Live logs in real-time
✅ Smart alerts built-in
✅ Auto-scaling without configs
✅ Cost monitoring included
✅ Everything in one place
The goal isn't to replace every tool.
It's to replace the need for most of them.
Curious what a unified dashboard actually looks like?
We built it to feel familiar, not overwhelming → https://kuberns.com
How many tools are you switching between daily?
#DevOps#BuildInPublic#DeveloperTools#StartupIndia#LinkedInNews
🔇 Apache Airflow 3.X: What's New, What Hurts, and Should You Upgrade?
Spent some time deep-diving into Airflow 3.X, and honestly - this release changes a lot.
From the React UI and event-driven scheduling to “run anywhere” tasks, it’s Airflow finally catching up to modern data teams - but not without a few gotchas.
Check out my latest article to see if it’s worth the jump 👇
#DataEngineering#ApacheAirflow#ETL#DevOps#CloudApache AirflowAstronomerDanube Data Labs
🌩️ Weekend Deep Dive – AWS Event Bridge Pipes (The Future of Serverless Integration)
🚀 #AWSTip — The most underrated AWS service in 2025: Event Bridge Pipes
Most engineers still connect services using Lambda glue code, but there’s a smarter way… 😍
👉 EventBridge Pipes can directly connect your event sources (like SQS, Kinesis, DynamoDB Streams) to targets (like Step Functions, Lambda, EventBus) —
without writing a single line of custom code.
💡 Why it matters:
✅ Less code → Fewer bugs
✅ Lower cost → No extra Lambda triggers
✅ Faster flow → Native event streaming
✅ Built-in filtering, transformation, and retries
With Pipes, you can replace messy integrations with a clean, serverless data pipeline that’s secure, scalable, and maintainable.
⚙️ Example use case:
When a new message arrives in SQS → EventBridge Pipe → triggers a Step Function workflow → processes the event automatically.
No Lambda. No cron jobs. No delay.
This weekend, try setting up your first Pipe — and you’ll never design event-driven architectures the same way again. 💪
#AWS#EventBridge#Serverless#DevOps#Automation#CloudEngineering#WeekendLearning#CloudWithChinmay
💡 Sometimes, the fix isn’t in code — it’s in the one button you always scroll past.
Last night, my Azure Data Factory publish kept failing with the classic: “Unable to delete 'azuresql_ls' because it’s being referenced by 'Incremental Ingestion’...” I went through every branch, every linked service JSON, every Git merge — nothing worked. Then I realized: the issue wasn’t my pipeline; it was my Git–Live sync being out of phase.
The solution? Overwrite Live Mode. One overlooked option that instantly fixed the stale publish branch and restored both linked services.
What I learned:
- ADF doesn’t store non-AKV secrets in Git; it pushes them straight to Live.
- Case sensitivity matters — AzureSql_ls ≠ azuresql_ls.
- When Live and Git get misaligned, “Overwrite Live Mode” resets the truth source.
Debugging cloud pipelines teaches you more about DevOps discipline than tutorials ever will.
Have you ever hit an ADF or CI/CD issue that made you question your entire setup? Drop it below — I’m collecting “Data Factory war stories.” 👇
#AzureDataFactory#DataEngineering#MicrosoftFabric#DevOps#GitIntegration#LearningByDoing
