Qdrant’s Post

🚀 𝐇𝐨𝐰 𝐖𝐞 𝐌𝐚𝐝𝐞 𝐒3 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 80% 𝐅𝐚𝐬𝐭𝐞𝐫 (𝐚𝐧𝐝 𝐂𝐡𝐞𝐚𝐩𝐞𝐫!) When our boto3 scripts started slowing down under millions of S3 files, we knew it was time to rethink ingestion from the ground up. 💡 Working with Rohan Giriraj, we rebuilt our entire approach using S3 Batch Copy — shifting ingestion from compute-heavy scripts to AWS’s managed orchestration. The result: ⚙️ 100K+ files/min throughput 💰 ~60% lower cost (no EC2 retries!) 🌍 Cross-region copies made native 📈 Zero manual retries — fully managed, fully scalable We learned that boto3 gives flexibility, but Batch Copy gives velocity — and the peace of mind that comes with it. 👉 Read full story on Medium: https://lnkd.in/gq7aCQg6 #DataEngineering #AWS #S3 #BatchCopy #DataPlatform #AutodeskTechTrailblazers #Ingestion #BigData #Automation

Introducing AlacticAGI - The Enterprise Framework for AI Data Infrastructure. Built to simplify dataset creation, indexing, and orchestration for LLMs and intelligent systems. Unified Python architecture Async-native pipelines Built-in observability Cloud-ready for AWS, Azure & GCP Available Beta Version now on PyPI: pip install alactic-agi Docs: https://docs.alactic.io Download: https://lnkd.in/gTiaHPi2 Revolutionizing how AI systems learn from the world. It’s great to announce our very first product ever.

A GitHub repo that contains everything related to MLOps. • Week 0: Project Setup • Week 1: Model Monitoring - Weights and Biases • Week 2: Configurations - Hydra • Week 3: Data Version Control - DVC • Week 4: Model Packaging - ONNX • Week 5: Model Packaging - Docker • Week 6: CI/CD - GitHub Actions • Week 7: Container Registry - AWS ECR • Week 8: Serverless Deployment - AWS Lambda • Week 9: Prediction Monitoring - Kibana If you're serious about deploying ML models in the real world, this is a great place to start. #MLOps https://lnkd.in/gtzth7uX

Every once in a while, I come across a performance deep dive that answers a question I’ve always wondered about — and just keeps going. This time it’s Ben Chess’s incredible blog post on scaling Kubernetes to 1,000,000 nodes. He digs into every layer of the problem — from running out of IPs to pushing the limits of etcd and API performance — and explains how he overcame each bottleneck. If you love performance engineering, distributed systems, or just great technical storytelling, this one’s worth your time: https://lnkd.in/dAJn9sfw

Learn to deploy AI Agents on AWS +6 hours of hands-on content 👇 In this +6 hour workshop, I'll take you from the very fundamentals all the way to a production-ready Agent deployed on AWS. ➤ 𝐏𝐡𝐚𝐬𝐞 𝟏: 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐭𝐡𝐞 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧 We'll start with: 1️⃣ Fundamentals of Agents and Agent architectures 2️⃣ Everything you need to know about LangGraph 3️⃣ 7 hands-on coding labs to put theory into practice ➤ 𝐏𝐡𝐚𝐬𝐞 𝟐: 𝐆𝐨𝐢𝐧𝐠 𝐭𝐨 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 𝐨𝐧 𝐀𝐖𝐒 Once the fundamentals are solid, we’ll deploy together: 1️⃣ Use GitHub Actions to push your app to AWS ECR 2️⃣ Build Lambda functions from our containerised application 3️⃣ Add API Gateways and create webhook endpoints 4️⃣ Add observability with Opik (by Comet) 5️⃣ Implement memory systems with MongoDB and Qdrant 🧩 Watch both parts here: 👉 Part 1 → https://lnkd.in/eNsrfp_G 👉 Part 2 → https://lnkd.in/dQnY4i8y Would you like to see more workshops like this one? If so, what kind of applications should I cover next? 🔁 Reshare this to help more builders get out of notebooks! Let's build!

Build production RAG pipelines with 18 lines of YAML 🚀 RAG applications need data from various sources moved into vector stores. Manual API integration means writing boilerplate for rate limiting, pagination, and error handling instead of building AI. CloudQuery handles the entire data-to-embeddings pipeline with declarative YAML config and native pgvector support. Key benefits: • Pre-built connectors for AWS, GCP, Azure, and 100+ platforms • Sync state persistence with incremental processing and automatic schema evolution • Built-in PII removal, column obfuscation, and data cleaning for compliance • Native pgvector support: text splitting, embeddings, semantic indexing for RAG Plus, CloudQuery is open source! Install it with "pip install cloudquery". #DataEngineering #ELT #Colaboration #DataPipelines

You can build a production RAG pipelines with 18 lines of YAML with CloudQuery?!?! Checkout this great post to learn how!

Reducing Docker Image Size for a GenAI App on AWS EC2 While deploying a GenAI application that uses libraries like sentence-transformers, langchain, and transformers, Docker image size can grow beyond 6GB. This causes slow CI/CD pipelines, longer EC2 startup times and unnecessary ECR storage costs. We decided to optimize the Dockerfile and requirements.txt. Result: image size reduced from 6.3 GB to 2.1 GB and EC2 deployment time dropped from 8 minutes to under 3 minutes. Optimized Dockerfile: ----------------------------------------------------------- # Use minimal base image FROM python:3.12-slim WORKDIR /app COPY requirements.txt . # Install only what's required and clean up cache RUN apt-get update && apt-get install -y --no-install-recommends git \   && pip install --no-cache-dir -r requirements.txt \   && rm -rf /var/lib/apt/lists/* # Copy source code last for better layer caching COPY . . CMD ["python", "app.py"] ------------------------------------------------------- Additional steps 1. Use .dockerignore to exclude local environments, cached models, and unnecessary files. 2. Load large models (e.g., Sentence Transformers) dynamically from Hugging Face Hub instead of bundling them inside the image. 3. Reduce redundant dependencies in requirements.txt by merging overlapping langchain sub-packages. Special thanks to ByteByteGo, Nana Janashia, Krish Naik and their videos on Docker optimization, CI/CD practices and Generative AI which helped me learn these concepts. #GenerativeAI #Docker #AWS #EC2 #MLOps #LangChain #Python

𝗛𝗼𝘄 𝘁𝗼 𝗮𝗱𝗱 𝗠𝗖𝗣 𝗦𝗲𝗿𝘃𝗲𝗿𝘀 𝘁𝗼 𝗢𝗽𝗲𝗻𝗔𝗜’𝘀 𝗖𝗼𝗱𝗲𝘅 𝘄𝗶𝘁𝗵 𝗗𝗼𝗰𝗸𝗲𝗿 𝗠𝗖𝗣 𝗧𝗼𝗼𝗹𝗸𝗶𝘁 https://lnkd.in/eG7Yh-GY AI assistants are changing how we write code, but their true power is unleashed when they can interact with specialized, high-precision tools. OpenAI’s Codex is a formidable coding partner, but what happens when you connect it directly to your running infrastructure? Enter the Docker MCP Toolkit. The Model Context Protocol (MCP) Toolkit acts as a secure bridge, allowing AI models like Codex to safely discover and use any of the 200+ MCP servers from the trusted MCP catalog curated by Docker.

Ever felt like your Kubernetes cluster's logs are a chaotic mess, scattered across pods and hard to search? Let's fix that with Loki , the Cloud Native Computing Foundation (CNCF)'s lightweight log aggregation system inspired by Prometheus. It's designed for scale without the bloat of traditional log solutions. Loki indexes labels rather than log contents, making it super efficient for querying massive volumes of logs. Pair it with Promtail (its agent) to scrape logs from your pods and send them to a central store. This means you get full-text search across your entire cluster without breaking the bank on storage. Practical use case: Imagine debugging a microservice outage. Instead of SSHing into nodes or tailing individual pod logs, fire up Grafana (Loki's go-to dashboard), query by service name or namespace, and spot the error patterns instantly. For example, set up a simple Promtail config to tail container logs and label them with Kubernetes metadata like pod name and labels. Loki stores them in object storage like S3 for durability. To get started: 1. Install Loki via Helm: helm install loki grafana/loki 2. Deploy Promtail as a DaemonSet to collect logs. 3. Connect to Grafana and create dashboards for log exploration. This setup not only saves time but integrates seamlessly with your existing Prometheus/Grafana stack. No more log silos—unify your observability! Pro tip: Use structured logging in your apps (JSON format) to make queries even more powerful with LogQL. What's your go-to logging tool in K8s? Share below! #CNCF #Kubernetes #Observability #DevOps