Dagster Labs

Heading to Small Data SF? Don't miss this hands-on workshop 🛠️ Dennis Hume, our Developer Advocate, is teaching "Composable Data Workflows: Building Pipelines That Just Work" and you won't want to miss it. Here's what makes this workshop different: you'll build a real system that identifies duplicate GitHub issues. No external services, no authentication, zero cost. Just you, your code, and a GitHub Codespace. What you'll learn: 🐙 Data orchestration fundamentals with Dagster 📦 Working with partitioned assets and resources ✅ Implementing data quality checks that actually work 🦆 Vector similarity search with DuckDB ⚙️ Building automation with schedules and sensors ⚒️ Practical embeddings for real-world deduplication This is a hands-on workshop—no slides-only theory here. You'll leave with working code and patterns you can actually use.

🆕 From BI Sprawl to Context Stores: How Teams Actually Ship AI 🏆 How Dagster replaced 80% of ad-hoc BI dashboards with a new Slack-native data analysis tool (a conversation with Nick Schrock of Dagster Labs) https://lnkd.in/gnAPjCvH

Dagster 1.12 so called "Monster Mash" shipped. The UI is cleaner. Collapsible sidebar frees up space, navigation is faster, everything you need is easier to find. Components are now GA with standardized interfaces across integrations. StateBackedComponent lets you persist state separately from config. New integrations for Census, Cube, Hex, Mode, and Preset. FreshnessPolicies hit GA. Deploy scaffolding actually works (dg scaffold generates Docker + GitHub Actions configs). Time-based partitions support exclusions. Backfills accept run config. We also launched a Support Center and reorganized the docs. New examples for DSpy, PyTorch, and advanced patterns. Read the full release blog today!

Delaney Housley is our recruiting manager. She used to spend 15+ hours a week in LinkedIn searches. She doesn't anymore. In this demo, she shows how she uses Compass to source engineers with hyper-specific criteria in minutes. Company stage, graduation years, internship experience, funding status. The kind of filters that normally mean opening 30 tabs and losing your mind. Three prompts. Three minutes. Full candidate list ready to export. This is what we built Compass to do: turn sourcing from a multi-day grind into something you can knock out between meetings. Check out her full video on our YouTube and our demo instance in our community slack. Links in the comments!

Hello everyone! I've been on the looking for a new ETL/data orchestration tool for a while, and I finally stumbled upon something truly unique: Dagster. It's open-source, remarkably easy to use, and what really blew me is its asset-centric approach. This unique perspective lets you see all your data assets and their lineage in a single view. Imagine being able to classify and visualize your data flow into a familiar pattern like the Medallion Architecture (Bronze, Silver, Gold), right in the lineage graph. I'm so impressed that I've started building resources to help others dive in... I'm creating videos and unofficial documentation, and I plan to keep adding tutorials on all the core modules and components. Check out the resources below and let me know what you think of Dagster's approach! YouTube Playlist (Videos & Tutorials): https://lnkd.in/d8ScEzC3 Unofficial Documentation: https://lnkd.in/drGTQCau Dagster Labs #DataEngineering #ETL #Dagster #OpenSource #DataOrchestration #DataLineage

Our Software Engineer Colton P. just published on the OpenAI blog about something we accidentally discovered while overhauling our documentation process: clear structure doesn't just help humans contribute, it makes AI exponentially more useful too. Context is everything. Having code, docs, and examples in a monorepo isn't just convenient—it gives AI the full picture. No more vague, hallucinated explanations. Point Codex to a PR with proper context, and it actually understands what changed and why. Documentation as structured data. When your CONTRIBUTING.md clearly outlines hierarchy, structure, and best practices, you're not just helping contributors; you're creating a map that AI can follow. Codex writes better documents when we write better guidelines. Different audiences want different formats (blog posts vs. video scripts vs. tutorials), but the core ideas stay the same. We're using Codex to translate between mediums, saving hours of rewriting while keeping our voice consistent. Read Colton's full post on the OpenAI blog. Link in the comments!

Super excited for my post to be live on OpenAI's developer blog: "Using Codex for education at Dagster Labs". At Dagster Labs we make heavy use of AI tooling across the company, and that includes for educational content: creating, reviewing, and auditing, and learning how things work. In this post I outline some of the lessons we've learned. This includes: * The importance of CONTRIBUTING.md files * The (controversial) advantages of using a mono repo * How we use Codex to learn about core framework changes * And more! Would love to hear what y'all think! https://lnkd.in/eD9kFm8U