Sandeep Sukhani

Worth giving your 2 minutes..

6

𝗡𝗲𝘅𝘂𝘀 & 𝗝𝗙𝗿𝗼𝗴: 𝗥𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝗶𝘇𝗶𝗻𝗴 𝗗𝗲𝘃𝗢𝗽𝘀 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 🚀 "Day 3 of 30 days 30 Java-based tools" In the world of 𝗗𝗲𝘃𝗢𝗽𝘀 and 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻/𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝗹𝗶𝘃𝗲𝗿𝘆 (𝗖𝗜/𝗖𝗗), tools like 𝗡𝗲𝘅𝘂𝘀 and ��𝗙𝗿𝗼𝗴 𝗔𝗿𝘁𝗶𝗳𝗮𝗰𝘁𝗼𝗿𝘆 are game-changers! These platforms empower developers and organizations to manage, secure, and scale their software delivery pipelines with ease. Why Nexus & JFrog Matter: ✅ 𝗔𝗿𝘁𝗶𝗳𝗮𝗰𝘁 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁: Centralized repository for storing and managing builds, dependencies, and container images. ✅ 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗙𝗶𝗿𝘀𝘁: Advanced vulnerability scanning ensures your software is safe and compliant. ✅ 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗕𝗼𝗼𝘀𝘁: Reduce build times with local caching and efficient dependency resolution. ✅ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Perfect for organizations aiming to scale their DevOps practices globally. Whether you're a startup looking to streamline deployments or an enterprise aiming for reliability at scale, 𝗡𝗲𝘅𝘂𝘀 and 𝗝𝗙𝗿𝗼𝗴 𝗔𝗿𝘁𝗶𝗳𝗮𝗰𝘁𝗼𝗿𝘆 provide the foundation for seamless software delivery. 👉 Your Turn: ➔ Are you using Nexus or JFrog in your CI/CD pipelines? ➔ What’s your favorite feature, or what challenges have you faced? Drop a comment and let’s share insights! #DevOps #CICD #Nexus #JFrog #SoftwareEngineering #ArtifactManagement

7

3 Comments

Open sourcing "Kube Watchtower"... My initial idea was to create an open-source alternative to k8s lens. v0.1.0 is marked as pre-release. Features include - 1. An overview of the entire Kubernetes cluster. 2. Live logs of the pods. 3. Quick overview of deployments, services, and config maps. 4. Real-time view/update/scale of deployments and config maps. 5. Single binary with embedded UI resources. 6. Fully private AI assistance. #kubernetes #opensource #go

167

13 Comments

Fill “standards” in the blanks: Like #Kafka is for Streaming.. Like #S3 is for Object Storage… Like #PostgreSQL is for RDBMS… Like #Linux is for Operating systems… ……….. is for Monitoring… ……….. is for Containers… ……….. is for Cloud Deployment… ……….. is for Web APIs… ……….. is for Systems Programming… ……….. is for Game Development… ……….. is for Mobile Development… ……….. is for Unit Testing… ………. is for API Testing… ……….. is for Load Testing… ……….. is for ………… What will you add more?

9

3 Comments

Is'nt every backend exists to serve front end? otherwise why it would called so? So you are suggesting 2 types of back ends? 1. "new" back end to front end 2. traditional back end? Question to all ? Where is bulk of business logic ... Now ? Tracing the history from 2000's 1. VB/Oracle Desktop App : Front end 50%, Backend (DB) 50%. Stored procedures and PL/SQL code encoded business logic 2. Early Web Apps - Front end (ASP/JSP) - 20%, Middle Tier (App server) 50% and database (30%) 3. Modern Rich Internet apps, Front end (10% or less), Microservices (85%) and 5% in databases. What do you think?

2

For CKA Exam aspirants, There’s some good news and some bad news. The bad news is, The CKA exam is updating after November 25, 2024. New topics will be added, focusing more on real-world Kubernetes skills like the Gateway API, Helm, Kustomize, CRDs, and Operators. The good news is, These updates are great for those who want to truly master Kubernetes and become real-world CKA experts. I’ve written a blog comparing the old and new syllabus and outlining the additions. 𝗖𝗞𝗔 𝗨𝗽𝗱𝗮𝘁𝗲 𝗕𝗹𝗼𝗴: https://lnkd.in/gsqUc5TG 𝗡𝗼𝘁𝗲: Any CKA exam taken after the updated release will test on the new set of Domains and Competencies I personally wanted the syllabus to change because Kubernetes administration has evolved a lot in the real world. These updates will help engineers with the concepts needed for real-world setups. ♻️ PS: Dont forget to Repost and share this news with the community. #DevOps #Kubernetes #CKA

681

22 Comments

The best way to manage secrets for applications running on Kubernetes is to store your secret in a secrets manager like AWS Secret Manager or Hashicorp Vault and access the secret from the application using the SDK. The secret is fetched when the pod starts, stays in volatile memory (RAM) while the pod is running and gets removed once the pod is deleted. Kubernetes secrets is not the right way to store the secret. Except for RBAC mechanisms, there is no security. Anyone who gets master access to the cluster shall be able to access the secrets which provides a huge surface area for attack. Follow me for more such Devops and Kubernetes related content. #devops #kubernetes

5

#InnoDB, #MySQL's default storage engine, enhances performance on multi-CPU systems by supporting parallel query execution. This feature splits queries into sub-tasks, processes them concurrently across threads, and combines results for faster execution. #MySQLPerformance @MinervaDB #dba https://zurl.co/1txh

1

#InnoDB, #MySQL's default storage engine, enhances performance on multi-CPU systems by supporting parallel query execution. This feature splits queries into sub-tasks, processes them concurrently across threads, and combines results for faster execution. #MySQLPerformance @MinervaDB #dba https://zurl.co/1txh

8

✨ Engineering, specially deploying and managing applications at scale is a very big challenging task. 🤔 Managed Kubernetes - EKS, AKS and GKE only solves half of the problem of managing the control plane of Kubernetes. It's still the work of your Devops / SRE / Backend engineers to do all these - - Upgrade K8s and node versions frequently - Build, manage, update Helm Charts as needed - Maintain and monitor your application status, debug any node/infra/network failures - Build and maintain a full logging, monitoring stack Devops is costly, either hire experienced devops engineers, or make your backend engineers learn all this (developer time cost) 💰💰

61

2 Comments

AWS S3 Table feature.. lake house table format is converge to iceberg .. thoughts?

6

2 Comments

My Story of How I improved single server pod efficiency to handle 150 concurrent users operating at 100% cpu usage to 900 concurrent users operating at 85% cpu usage It has been a long day and a long time since I had posted so here's some of my recent venture into servers, server-side latency and infra costs. So the story starts with a load test request for one of our apis on a Friday morning looking forward to push a new feature to prod by evening. My mind goes well that's a double edged sword when it comes to critical requests on a Friday :) Anyways It's a simple request with a small twist to it since I wanted to publish the results for both cached and non-cached behavior of the same api. I build my jmeter scripts and run it on Azure Load Runner and bam first surprise.... As the number of virtual users increased, our poor little single pod taking up the huge loads suffered and the latency gradually went up until certain breaking point where it couldn't handle the said number of concurrent users. Surprisingly the same pod could handle same rps with way less number of users! This pod had a spec of 2 cpu and 4G memory. The server could handle merely 29 users at 191ms which is < 200ms (standards) and this seemed too low.  Moreover at ~150 users the latency shot upto 48s and at ~200 users it had reached 3mins! This led me into a rabbithole trying to debug what might be issue which led me into profiling the memory and cpu usage during such loads and long story short the cpu usage was pretty high i.e., 100% but low memory. After a live profiling of such pods via VisualVm I was able to conclude the following: 1. Number of threads must be improved. 2. Threads must be efficiently used i.e., less waiting time on threads. 3. Reduce the cpu usage. The solution for the above problem was: 1. Tomcat thread and queue management for best use of available resources. 2. Server change from BIO to NIO mode (Non-Blocking IO) 3. Connection pooling 4. Increase Heap memory and set min and max to handle throttling. Now the trick is to not over-do the number of threads and tasks handled by a thread since it can increase the memory usage again trggering gc events which is bad. After fine ballet of all above mentioned fixes lo and behold a single pod could handle  900 concurrent users 1k RPS at 950ms with cpu as low as 85% and low gc activity!! #springboot #jmeter #backend #infra #costcutting

12

How we solved Docker rate limiting issues on EKS and saved thousands of dollars in NAT Gateway costs ? Recently, we faced a challenge where we were running into two major issues for a batch processing workload for one of our clients. Here is an explanation of the scenario. We had to spin up hundreds of containers on demand on EKS for batch processing. 1. Hundreds of jobs were started at the same time. 2. The image used for batch processing was large in size ~ about 2.5 GB.  3. Dockerhub was used to store the images. 4. This activity of batch processing happens hundreds of times a day resulting in hundreds of containers being spun up every day. Here are the two challenges we were facing: 1. Dockerhub Rate Limits - We were hitting rate limits on pulling images from Dockerhub. Dockerhub allows 200 pulls per 6 hour period for Authenticated Users.  2. High NAT Gateway costs - It was causing a very high amount of traffic ingress through the NAT gateway which was causing huge NAT gateway bills. Even though AWS and GCP charge only for data transfer out, a NAT gateway charges for data flow in either direction. We had to stay on Dockerhub for the following reasons: 1. We had to support customers in multiple regions.  2. We also had to spin up the containers on Google Kubernetes Engine (GCP). Hence it made sense for our CI/CD to push images in one place. AWS ECR Pull through cache to the rescue AWS ECR Pull Through Cache is a feature provided by ECR that allows you to pull images from Dockerhub via ECR. It caches the images once the image is pulled from Dockerhub and continues to serve it from there. Lets say you have an image in Dockerhub that lives in the `<organization/repository>`. To leverage the ECR Cache, the only change that you need to do is 1. Create a Pull through cache rule in ECR.  2. Instead of pulling images from `<organization/repository>`, you need to start pulling images from `<accountnumber>.dkr.ecr.<region>.amazonaws.com/docker-hub/<organization>/<repository>` . Here are a few points: 1. In case the image does not exist in ECR, It will be pulled from Dockerhub and stored in ECR the first time the image is requested for.  2. The second time onwards, the image will be served from ECR.  3. The repository does not need to pre exist in ECR. It is created on the fly as the image is pulled. 4. You can store your Dockerhub credentials in AWS secretsmanager to be able to pull private images from Dockerhub.  5. Data transferred between ECR and other services in the same region is free of charge. Using ECR Pull through cache 1. Our NAT Gateway bills dropped instantly and it was visible in the cost explorer dashboard from the next day.  2. We were able to by pass docker rate limits and spin up hundreds of containers every day. PS: If you are looking for help in solving such challenges in your cloud infrastructure, let's connect. #kubernetes #aws #costoptimization

20

2 Comments

Here's an interesting Kubernetes topic. To schedule a pod, Kubernetes selects multiple feasible Nodes. These are the nodes that have met scheduling requirements. Later, set of functions are run on each node to score one node where a pod can be deployed. Now, if there are 1000s of nodes and to run functions on top of those can increases latency. Because evaluating every single node can be time-consuming. Particularly for larger clusters with 1000s of nodes. percentageOfNodesToScore. The percentage value can be between 0 and 100. Interestingly, if you configure anything above 100, kube-scheduler intelligently takes 100. This setting helps scheduler to prioritizes finding nodes that meet the pod's resource requirements. Once the percentage of feasible nodes is reached, it stops searching further. Optimizing performance. 🔁 Hope it's useful. Repost helps your friends.

206

9 Comments

Stripe’s ~300M metrics, 40k alerts and 100k dashboard queries produced by ~7k employees migration to AWS Scalability limits, reliability issues, and increased cost triggered re-architecture of metrics monitoring solution. # Challenges - Cost - scalability # HLD Stripe formulated a four phase approach to this migration 1. Dual-write metrics to the legacy time-series database (TSDB) and AMP 2. Translate assets (alerts and dashboards) from the source language into PromQL and Amazon Managed Grafana Dashboards. 3. Validate the results of the translated assets and the dual written metrics in an iterative fashion 4. Migrate users’ mental models to a Prometheus compatible mental model # How Metrics flows 1. Collection: Metrics are collected from host applications as well as scraped from Kubernetes clusters and sent to aggregation layer. 2. Aggregation: The shuffler delivers data to the correct host while the aggregator averages gauges, sums counters and computes percentiles for cardinality reduction. 3. Writing to AMP: Aggregation layer sends the metrics to the egress proxy that writes to AMP. Metrics from CloudWatch are also written to AMP through egress proxy to have a unified storage layer.. 4. Dual-Write: The aggregated metrics as well as unaggregated metrics are also sent to existing egress proxy to write the metrics to legacy Time Series Database TSDB during the migration process # Reference and Image Credit - aws .amazon .com/blogs/mt/how-stripe-architected-massive-scale-observability-solution-on-aws/ #SystemDesign #architecture #observability --- Understanding the First Principles is the key to Effective Learning! 🔔 Follow Kamran to Improve System Design Skills

65

Moving applications to https has become simpler than ever! Using letsencrypt and certbot, I was able to migrate https://ramayanquiz.com to https in less than 10 minutes. certbot makes the entire process seamless. With a single command, I was able to generate the certificate and instruct certbot to automatically configure nginx for https support. https://howhttps.works illustrates why https is crucial and how it works in an interesting way!

5

👉 𝐖𝐡𝐲 𝐦𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠 𝐚𝐧𝐝 𝐭𝐫𝐨𝐮𝐛𝐥𝐞𝐬𝐡𝐨𝐨𝐭𝐢𝐧𝐠 𝐊𝐚𝐟𝐤𝐚 𝐢𝐬 𝐬𝐨 𝐡𝐚𝐫𝐝? Working last few years in observability domain, we have had multiple users complain about the lack of detailed monitoring for messaging queues and Kafka in particular. Especially with the coming of instrumentation standards like OpenTelemetry, we thought there must a better way to solve this. 💡 We dived deeper into this problem and identified that leveraging tracing can be useful. Most current monitoring solutions for Kafka show metrics about Kafka, but metrics are often aggregated and often don’t give much details on where exactly things are going wrong. 𝐖𝐞 𝐰𝐨𝐮𝐥𝐝 𝐥𝐨𝐯𝐞 𝐭𝐨 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 𝐢𝐟 𝐭𝐡𝐢𝐬 𝐩𝐫𝐨𝐛𝐥𝐞𝐦 𝐬𝐭𝐚𝐭𝐞𝐦𝐞𝐧𝐭 𝐫𝐞𝐬𝐨𝐧𝐚𝐭𝐞 𝐰𝐢𝐭𝐡 𝐭𝐡𝐞 𝐜𝐨𝐦𝐦𝐮𝐧𝐢𝐭𝐲 𝐡𝐞𝐫𝐞 𝐚𝐧𝐝 𝐰𝐨𝐮𝐥𝐝 𝐥𝐨𝐯𝐞 𝐚𝐧𝐲 𝐟𝐞𝐞𝐝𝐛𝐚𝐜𝐤 𝐨𝐧 𝐡𝐨𝐰 𝐭𝐡𝐢𝐬 𝐜𝐚𝐧 𝐛𝐞 𝐦𝐨𝐫𝐞 𝐮𝐬𝐞𝐟𝐮𝐥 𝐭𝐨 𝐲𝐨𝐮. In the first problem statement for this series, we are focusing on 𝐂𝐨𝐧𝐬𝐮𝐦𝐞𝐫 𝐋𝐚𝐠. Possible reasons for an increase in consumer lag are: 1️⃣ 𝐒𝐥𝐨𝐰 𝐜𝐨𝐧𝐬𝐮𝐦𝐞𝐫𝐬: Slow consumers that take a long time to process messages can cause consumer lag. 2️⃣ 𝐌𝐞𝐬𝐬𝐚𝐠𝐞 𝐬𝐢𝐳𝐞: Large message sizes can cause consumer lag, especially if the consumer is not configured to handle large messages. 3️⃣ 𝐏𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐬 𝐠𝐞𝐭𝐭𝐢𝐧𝐠 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞-𝐜𝐨𝐧𝐬𝐭𝐫𝐚𝐢𝐧𝐞𝐝 with high memory or CPU usage. 4️⃣ 𝐇𝐢𝐠𝐡 𝐦𝐞𝐬𝐬𝐚𝐠𝐞 𝐭𝐡𝐫𝐨𝐮𝐠𝐡𝐩𝐮𝐭t: High message throughput can overwhelm consumers, causing consumer lag. 5️⃣ 𝐍𝐞𝐭𝐰𝐨𝐫𝐤 𝐥𝐚𝐭𝐞𝐧𝐜𝐲: High network latency between the Kafka cluster and the consumer can cause consumer lag. In this proposal, a user can see consumer lag graphs for for different Consumer Group - Partition ordered by average consumer lag. If she wants to find more details about a particular point ( which corresponds to a specific consumer group and partition), she can find more details about it in a side panel. 𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 𝐬𝐨𝐦𝐞 𝐰𝐢𝐫𝐞𝐟𝐫𝐚𝐦𝐞𝐬 𝐨𝐧 𝐩𝐫𝐨𝐩𝐨𝐬𝐞𝐝 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬, 𝐛𝐮𝐭 𝐭𝐡𝐨𝐬𝐞 𝐚𝐫𝐞 𝐣𝐮𝐬𝐭 𝐭𝐨 𝐩𝐮𝐭 𝐨𝐮𝐫 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐭𝐡𝐨𝐮𝐠𝐡𝐭 𝐩𝐫𝐨𝐜𝐞𝐬𝐬 𝐦𝐨𝐫𝐞 𝐜𝐨𝐧𝐜𝐫𝐞𝐭𝐞𝐥𝐲. 𝐖𝐞 𝐰𝐨𝐮𝐥𝐝 𝐥𝐨𝐯𝐞 𝐚𝐧𝐲 𝐟𝐞𝐞𝐝𝐛𝐚𝐜𝐤 𝐨𝐧 𝐰𝐡𝐚𝐭 𝐟𝐥𝐨𝐰𝐬, 𝐬𝐭𝐚𝐫𝐭𝐢𝐧𝐠 𝐩𝐨𝐢𝐧𝐭𝐬 𝐰𝐨𝐮𝐥𝐝 𝐛𝐞 𝐦𝐨𝐬𝐭 𝐮𝐬𝐞𝐟𝐮𝐥 𝐭𝐨 𝐲𝐨𝐮. If you want to check the complete blog, you can check out the linked blog in the comments.

49

10 Comments