Astronomer

What makes complex workflows easier to manage and scale effectively? Learn from Mihir Samant of Etraveli Group how Airflow empowers teams to tackle bottlenecks, automate tasks, and streamline finance operations. His insights highlight the simplicity and power of leveraging the right tools to drive efficiency. Click the link in the comments for the full episode. #AI #Automation #Airflow #MachineLearning

Many data leaders understand data orchestration is essential, but few know what it really costs. 📈 Engineering time, incident recovery, and tool sprawl all add up fast. That’s why we built the Astro TCO Calculator — a quick, self-service way to model the total cost of running Airflow and uncover where a unified orchestration platform delivers savings. 🏦 Learn your potential ROI in seconds: https://bit.ly/47AOYZV

“The type of agents we are talking about here don’t know what data tables are reliable, they don’t know which data pipelines are breaking, who owns what, or how changes ripple through the system. Unsurprisingly, the orchestration layer turns out to be where all this crucial context naturally lives. Every time a pipeline runs, fails, or succeeds, it’s creating a rich trail of metadata. Every data transformation, every quality check, every usage pattern gets logged. It’s like a comprehensive flight recorder for the entire data platform.” - Julian LaNeve, CTO, Astronomer Julian spoke with ComputerWeekly.com's Adrian Bridgwater about why the data orchestration layer is the key to providing the necessary context to make AI actually useful. 📰 Read the article: https://lnkd.in/gkEUGE3y

Airflow fuels investigative journalism at scale. In “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI,” we feature Zdravko Hvarlingov at the Financial Times, who unpacks multi-tenant Airflow, story-finding pipelines and ML-driven entity matching that turns 600-page disclosures into newsroom-ready insights. Click the link in the comments for the episode. #AI #Automation #Airflow #MachineLearning

Do you spend more time debugging Apache Airflow than building new pipelines? The data team at WeWork saw a 67% reduction in infra management time after switching to fully-managed Airflow. Join our webinar on November 6 with Kenten Danas and Ellen Smith (Granoff) to learn: 💰 The hidden costs of manually scaling, upgrading, and securing Airflow yourself ⌚ How much time can you save by offloading Airflow management? Hint: More than you think! 🙅 How to stop wasting hours debugging and start shipping faster Register: https://bit.ly/4hvhQqS

NEW VIDEO 🤩 The (real world) Reddit Project! 👇 Toy projects are cool, but... not super useful. This time, you will build a REAL pipeline that runs in production! This video covers the data pipeline we use at Astronomer to identify Reddit posts asking questions about Airflow or Astronomer. It: 1️⃣ Fetches the posts 2️⃣ Categorizes them using OpenAI 3️⃣ Filters the posts 4️⃣ Sends Slack notifications 5️⃣ Stores metrics in Snowflake You'll learn: ✅ Designing a complex data pipeline with multiple branches and tasks. ✅ Setting up Snowflake connections and creating tables using Airflow. ✅ Fetching data from the Reddit API. ✅ Using an LLM (like OpenAI or similar) directly within tasks (AI SDK). ✅ Implementing branching logic to skip tasks when no data is found. ✅ Filtering and transforming data. ✅ Sending automated Slack notifications upon pipeline completion. And more! Enjoy ❤️ P.S.: We answer the questions, not AI ;) Link in the comments below 👇 #airflow #apacheairflow #dataengineer #dataenigneering

🤝 Meet with us at Anyscale's Ray Summit 2025! Our team will be in San Francisco next week and can help you understand how Astro, our unified DataOps platform powered by Apache Airflow, ensures reliable and scalable delivery of your mission-critical data. Learn more: https://bit.ly/4hBzrxA

We just added /llms.txt to Astronomer's docs 🎉 Example: https://lnkd.in/evq4V3xg 📚 What is /llms.txt? It's an emerging standard that makes documentation AI-assistant friendly. It helps LLMs quickly navigate and understand your content without parsing messy HTML, navigation bars, and ads. /llms.txt is designed to coexist with current web standards: - /robots.txt tells crawlers what they can access - /sitemap.xml lists all pages for search engines - /llms.txt provides curated markdown for LLMs It's designed for inference: real-time assistance when you're actually coding. When a developer asks their AI assistant about Airflow concepts or includes docs as context in their project, /llms.txt ensures the AI gets exactly what it needs: clean, structured, and relevant. 📌 Why data engineers should care: You can now instantly pull clean markdown versions of our docs as context. No more feeding it noisy HTML or incomplete snippets. Building a DAG and need context on dynamic task mapping or DAG Factory? Your AI now gets clean context, no noise, fast. 📔 For documentation maintainers: If you own developer docs, this is worth implementing. Check out llmstxt.org .. the setup is straightforward, but the impact is real. As AI becomes embedded in every developer's workflow, your docs need to serve both humans and their AI assistants. We're not just optimizing for Google anymore. We're optimizing for the tools developers actually use to ship code. #DataEngineering #ApacheAirflow #AI #DeveloperExperience #LLMs

🚀 Looking for guidance on how to scale Airflow with confidence? Over 80,000 organizations, including data teams at OpenAI, Uber, and Ford Motor Company, rely on Apache Airflow for data orchestration. Join Marc Lamberti for this live session packed with proven best practices to help you scale and run Airflow like an expert. You’ll learn how to: ✅ Optimize Airflow configurations ✅ Automate infrastructure management ✅ Maintain the health of your Airflow environments Register: https://lnkd.in/gDC253PR