Ward Bekker

🚨 Snowflake's Cortex 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌 function now supports strongly-typed outputs! We had structured outputs but the numbers still come back as strings, booleans as text, and so on. This meant adding 𝘛𝘙𝘠_𝘊𝘈𝘚𝘛 and other validation steps downstream. But not anymore! ✅ Now, you can enforce a strict schema with 𝗧𝘆𝗽𝗲 𝗟𝗶𝘁𝗲𝗿𝗮𝗹𝘀 directly in the 𝘰𝘶𝘵𝘱𝘶𝘵_𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦. This isn't just a hint to the model; 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌 now verifies each generated token against your definition to guarantee the response conforms to the specified types. 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁 𝗶𝘀 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰, 𝗿𝗲𝗹𝗶𝗮𝗯𝗹𝗲, 𝗮𝗻𝗱 𝗰𝗹𝗲𝗮𝗻 𝗱𝗮𝘁𝗮, every single time. You can now define your desired output using familiar SQL types. Just start your type literal with the 𝘛𝘠𝘗𝘌 keyword and use a SQL 𝘖𝘉𝘑𝘌𝘊𝘛 as the top-level structure. 🎯 This simple change has a massive impact: 1. 𝗥𝗲𝗱𝘂𝗰𝗲𝘀 𝗣𝗼𝘀𝘁-𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Your data lands correctly typed from the start. No more cleanup. 2. 𝗜𝗻𝗰𝗿𝗲𝗮𝘀𝗲𝘀 𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Eliminates an entire class of data quality errors, making your pipelines more robust. 3. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻: Perfect for systems that require deterministic, schema-compliant responses. This feature is supported by every model in 𝘈𝘐_𝘊𝘖𝘔𝘗𝘓𝘌𝘛𝘌. This is a huge leap forward for building 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗴𝗿𝗮𝗱𝗲, 𝗔𝗜-𝗻𝗮𝘁𝗶𝘃𝗲 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀. You can now build with more confidence and less defensive code. ⁉️ What other quality-of-life improvements are you using today? Pavan Mona Vivek Jessie Paritosh Boxin Nicole Jiwoong Arun Siddharth James Josh #Snowflake #DataEngineering #AI #LLM #SQL #SnowflakeCortex #Developer #DataQuality

635

20 commentaren

Everyone wants a “lightweight” data stack. But is it the right choice? It usually starts simple: Airbyte or Kafka for ingestion, S3, and even something like Iceberg for storage. Then you add Flink for processing, Airflow for orchestration, dbt for transformations… Suddenly, you’ve rebuilt a warehouse with a dozen services and no UI. This is exactly how stack bloat happens: 1️⃣ Each tool solves a narrow problem. 2️⃣ Integration work shifts to your engineers. 3️⃣ Operational overhead climbs with every new component. 👉 The result isn’t flexibility, it’s fragility across your whole stack. The wrong way to approach this is “warehouse or no warehouse”; instead, you should consider how much complexity your team is willing to own.

513

42 commentaren

🚀 New in Docling: Structured Data Extraction from Documents! 🚀 We’ve just added a brand-new functionality: extraction of structured data from complex documents using free-form schemas 🤩 What does that mean? 🔹 You can now skip the conversion step — instead of turning documents into text or JSON first, Docling directly extracts the structured fields you care about. 🔹 The requested fields are defined in a free-form schema, so you can instantly align the extraction with the schemas of your own databases. 🔹 This makes it ideal for data-pipelines that don’t need full document conversion, but do need to populate structured databases from messy, unstructured documents. ✨ It’s: 1️⃣ Super simple to use (check out the code snippet 👉 https://lnkd.in/eK6vaMEe ) 2️⃣ 100% open-source and runs fully local – no API calls needed 🙌 3️⃣ Powered by cutting-edge models from NuMind (YC S22) 4️⃣ Perfect for data-pipelines where you need to populate structured databases from documents (think invoices, Curriculum Vitae, contracts, product datasheets, etc) 5️⃣ Currently focused on PDFs and images (PNG) — support for pure text coming soon! The example below shows how easy it is to define a schema and extract structured fields directly from an invoice. 👉 Try it out, break it, send us feedback, and if you like what we’re building, don’t forget to ⭐ the repo (https://lnkd.in/d4UT-6_2)! After a long and fruitful summer, the Docling team has been cooking up many new features — and this is just the beginning! 🌟 #opensource #AI #documentAI #docling #IBM #IBMResearch

1.382

83 commentaren

Spark is overkill for most data work. DuckDB and PyArrow handle 80% of what you use Spark or DBT for. DuckDB transforms your data with pure SQL. PyArrow moves it across layers and formats like it’s nothing. No cluster. No scheduler. Just a laptop running production-grade pipelines. The stack is shifting —> back toward simplicity. #duckdb #dataengineering #pyarrow

473

83 commentaren

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 commentaren

🔊 Audio just became AI-native in Snowflake. AI_TRANSCRIBE is now available in Cortex — meaning you can transcribe audio files, detect speakers, and get word-level timestamps without managing any ML infra. Combine it with AI_SENTIMENT or AI_CLASSIFY and suddenly every call, podcast, or voice note becomes searchable, labelable, and insight-ready. I made a new Quickstart that walks through exactly how to set this up — including combining transcription with text and image analysis: 👉 https://lnkd.in/gyxeDdPi Can't wait to see how folks use this in production 🔥 #Snowflake #CortexAI #AITranscribe #MultimodalAI #GenAI #VoiceAI

112

5 commentaren

Enterprise data pulls in two directions. On one side: governed BI workloads — structured, compliant, and built for business reporting. On the other: unstructured pipelines — streaming feeds, heavy transformations, and machine learning at scale. Most organizations now run Snowflake and Databricks in parallel to handle this dual reality. Yet stitching them together creates new challenges: deciding which workloads belong where, moving data without waste, and keeping governance consistent across environments. Each platform brings distinct strengths to the table — Snowflake excels at governed analytics and BI reporting, while Databricks powers heavy transformations, streaming, and machine learning. When combined thoughtfully, these strengths converge into a unified architecture that supports the full data lifecycle. > Watch the on-demand discussion on how Janus Henderson Investors designed a unified architecture that balances both worlds: https://lnkd.in/eG-8YBqm In a financial services setting where accuracy, performance, and cost discipline matter most, Janus Henderson’s approach connects Snowflake’s strengths in analytics and governance with Databricks’ capabilities in engineering and ML. #unifieddataarchitecture #snowflake #databricks #usecase

135

38 commentaren

Pinot has gone through several evolutions 🌀 - From being append-only to supporting upserts - From a limited Pinot Query Language to Postgres SQL compliance and multi-stage query execution. - From tightly-coupled storage & compute to hybrid storage tiering options. And now, we’re taking our evolution one step further — native support for querying Parquet, unlocking the ability to serve directly on Iceberg! 🧊 🍷 Read on to understand why this is such a challenging feat for low-latency serving systems https://lnkd.in/ggWbbyzE 📖 We’re excited to share how we’re making it happen: abstractions for treating Parquet as the forward index, building indexes on top of it, and combining this with our tiered storage strategies of index pinning, block reads, and prefetching ⚙️ 🛠️ 🎨 Huge kudos to the team for the massive ongoing effort that makes this possible! 🙌

165

3 commentaren

Long Live Spark, Down with Spark 🙃 Spark has been the de facto processing engine for the last 10 years in the "Big Data" community. It still has its place in data processing. Some have branched off their own versions, spun off custom engines, and developed new approaches in an attempt to overcome the shortcomings of spark in diverse landscapes. Our focus is to enable the right compute for the problem with a focus on interoperability with other systems and to have open source compute engines as first-class citizens in platform. Enterprises are highly complex, with high heterogeneity of solutions, tech debt, data gravity, and modalities. I would argue that most pipelines in enterprise are not web scale; they are small and medium sized. A pipeline with a few million or hundreds of millions of rows, sometimes a few billion. Spark is overkill for 90%+ of these workloads and carries a lot of overhead that impacts speed and cost. With the Palantir Multi-Modal Data Plane (MMDP), we provide the framework to store data anywhere, compute data anywhere, and use any model. If we break down this a bit more, this means I can have data spread across different platforms, file formats, structures and use any compute (inside or outside of the platform) to process my data. The permutations of storage locations, formats, and compute engines are massive. It's the right tool for the problem at hand, not one tool you need to shove everything into. Re-platforming should never be the goal. With new frameworks like Polars, DataFusion, DuckDB and the availability of larger node sizes that this means you can run increasingly bigger and more complex workloads in a fraction of the time and cost. You can mix and match these as needed in Foundry and AIP, even having one step run one engine and another step use a different engine. (You can even bring your own engine.) Spark is always there on the table, but it should no longer be the default for most enterprises to start with.

232

24 commentaren

🚀🚀🚀 Big News just in!!! 🚀🚀🚀 📢 OpenAI open-weight 20B and 120B models (gpt-oss) are now available in ANZ and ASIA Geos on Databricks - served as PPT, PT, and Batch. 🚀 🤔 Why does this matter? 🌏 Because data sovereignty is NOT just a buzzword - it is a non‑negotiable for many organisations building with AI. By supporting Geo processing of workloads, customers can confidently develop secure, domain-specific AI agents without compromising compliance or performance. ⭐️ Just a few weeks ago, we shared how you can build intelligent, domain-specific AI agents powered by gpt-oss on Databricks. Now, with Geo processing, our customers can move even faster on their AI journeys. 🔎 Curious about how this unlocks new possibilities for your team? 👉 Dive in here: https://lnkd.in/gHQ45YEv 💬 Always happy to chat more - reach out if you would like to explore how this impacts your AI strategy! #Databricks #OpenAI #OpenWeightModels #DataSovereignty

79

Databricks' has two fairly new account offerings: The "Free Edition" & "Trial Accounts". Both are good for learning how to use Databricks, but they have different intended audiences. In this brief video, Greg Kroleski from Databricks' "Money Team" explains some of the key differences between these two offerings in a very practical way. Link: https://lnkd.in/eERbd7Ni

39

1 commentaar

Databricks - Managed table vs External table Any of the table consists of 2 parts. Table = Data + Metadata in managed tables both data and metadata are fully managed by databricks. Metadata is stored in the metastore. your data will be stored in your cloud storage. Basically for keeping data we need to create a external location and then mention that while creating the catalog. you might ask what is a external location. An external location is a cloud storage location backed by storage credentials to make it fully secured. Now this external location path you need to mandatorily specify at catalog level, if not specified at the metastore level. Databricks recommends not to give the external location path at metastore level as data from multiple workspaces will end up in same location. Now when we are creating schema, that time its optional to specify the external location and in case if we give that, this takes precedence. so lets take a scenario, if the external location is defined at all 3 metastore, catalog, schema then the path mentioned at schema level takes the precedence. Basically external location defined at lower level comes into effect. The managed table by default will be Delta format, and we could have created a iceberg table also. you might remember the days of hive metastore where when we drop the managed table both the data and metadata is dropped and there is no way to recover the data. In case of Unity catalog backed Managed table, after dropping the table we can write a UNDROP command within 7 days. so basically data is marked for deletion only after 7 days. However metadata is dropped. This makes sure that if a managed table is deleted by mistake, we can still undo things. Lets talk about External table The only difference in creation is, we give the external location path when creating a table. unity catalog governs data access permissions but does not manage data lifecycle, optimizations, storage location or layout. when you drop an external table, the data files are not deleted. when you create an external table, you can either register an existing directory of data files as a table or provide a path to create a new directory. Databricks recommends using external tables for the following use cases: - you need to register a table backed by existing data that is not compatible with unity catalog manage tables. - you also require direct access to the data from non databricks clients. So whenever possible try using the Managed tables, as you get all sorts of optimizations. Databricks recommends using Managed tables whenever possible. I am sure you would have learned something new from this post. Also, I recently launched my Databricks Masters Program. I have taken it as a challenge that this program should be the best. It will be a 15 weeks of extensive program (8 weeks of Databricks + 7 weeks of Databricks performance tuning) DM to know more!

99

11 commentaren