Pawel Potasinski’s Post

Did you know you can use SQL to directly query files on Databricks? With `read_files()` you can: • Query CSV, JSON, XML and more • Infer the schema • Combine with the rest of your data lake It's so powerful and it's one of the first things I show to any new user of our data platform, including engineers, analysts and business users. It immediately helps to show them that Databricks is different, and many things that they thought were impossible are suddenly simple. As much as we might want to, we can't avoid that some data will always live in files/sheets/excels. With `read_files` we can accept the file is the data source, but move the processing into SQL. Now you can: • Join the file with your data lake, instead of exporting more data to files • Leverage the power of SQL (particularly Databricks SQL) which can be more easily shared and reviewed by your team, instead of obscure and broken formulas and macros. Tips for success: • Setup your catalogs to provide users with easy access to Volumes, simplifying the file management and query experience. • Treat the file as any other CTE • Use the hidden `_metadata` column for data lineage • Use wildcard paths to read multiple files #databricks #datalake #sql

Being a Data Engineer, I often hear — “SQL is outdated.” But the truth? Most of the complex data pipelines and architectures still run on SQL. From Hive, Spark SQL, Flink SQL, BigQuery, Snowflake, Databricks to almost every modern data platform — SQL is the backbone. Languages evolve, frameworks change, but SQL remains the common ground that powers analytics, transformations, and decision-making at scale. So no — SQL is not dead. In fact, it’s more alive than ever. #sql #dataEngineer #dataanalytics #bigdata

Speed Matters When You are Working with Data. If you use Snowflake concatenate, applying the right habits can save time and keep pipelines reliable. Here are three tips from our new article: ·      Replace nulls with placeholders to keep outputs complete ·      Build composite keys that simplify joins across systems ·      Add clear separators so results stay readable in reports These are only a preview. Our full article shares more strategies for working with MongoDB, PySpark, and Power BI using Snowflake concatenate. Read here: https://lnkd.in/dweJvthx #Snowflake #SQL #DataEngineering #BigData #ETL #DataAnalytics #MongoDB #PySpark #PowerBI #CloudData #DataWarehouse #DatabaseManagement #DataScience #MachineLearning #DataStrategy #BusinessIntelligence #EnterpriseData #Xperti 

🐢 Slow queries aren’t always about hardware — sometimes, it’s about understanding 🔥 𝙃𝙤𝙬 𝘿𝙖𝙩𝙖𝙗𝙧𝙞𝙘𝙠𝙨 𝙎𝙌𝙇 𝙒𝙖𝙧𝙚𝙝𝙤𝙪𝙨𝙚𝙨 '𝙨𝙘𝙖𝙡𝙚' 𝙪𝙣𝙙𝙚𝙧 𝙡𝙤𝙖𝙙. Here’s another interesting practice exam question that dives into 𝗦𝗤𝗟 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹𝘀 — from query concurrency and scaling ranges to how the 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗹𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗲𝗿 𝗮𝗻𝗱 𝗮𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗲𝗿 handle queued workloads. 𝗤𝗨𝗘𝗦𝗧𝗜𝗢𝗡: You have noticed that Databricks SQL queries are running slow. You are asked to look reason why queries are running slow and identify steps to improve the performance. When you looked at the issue you noticed all the queries are running in parallel and using a SQL endpoint [ SQL Warehouse ] with a single cluster. Which of the following steps can be taken to improve the performance/response times of the queries? ⚠️ [ Option 1 ] They can turn on the Serverless feature for the SQL endpoint [ SQL warehouse ]. ✅ [ Option 2 ] They can increase the maximum bound of the SQL endpoint (SQL warehouse)’s scaling range. ⚠️ [ Option 3 ] They can increase the warehouse size from 2X-Small to 4XLarge of the SQL endpoint [ SQL warehouse ]. ❌ [ Option 4 ] They can turn on the Auto Stop feature for the SQL endpoint [ SQL warehouse ]. ❌ [ Option 5 ] They can turn on the Serverless feature for the SQL endpoint [ SQL warehouse ] and change the Spot Instance Policy to “Reliability Optimized.” This question is a perfect way to test: 1️⃣ How 𝗦𝗰𝗮𝗹𝗶𝗻𝗴-𝗨𝗽 𝘃𝘀 𝗦𝗰𝗮𝗹𝗶𝗻𝗴-𝗢𝘂𝘁 actually works inside Databricks SQL Warehouses. 2️⃣ The role of the 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗾𝘂𝗲𝘂𝗲, 𝗹𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗲𝗿, 𝗮𝗻𝗱 𝗮𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗲𝗿 in distributing query load efficiently. I’ve broken this down in an 11-minute explainer video — where I walk through the 𝘦𝘹𝘢𝘤𝘵 𝘧𝘭𝘰𝘸 𝘰𝘧 𝘩𝘰𝘸 𝘲𝘶𝘦𝘳𝘪𝘦𝘴 𝘨𝘦𝘵 𝘥𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘢𝘤𝘳𝘰𝘴𝘴 𝘤𝘭𝘶𝘴𝘵𝘦𝘳𝘴 and 𝘸𝘩𝘺 𝘩𝘰𝘳𝘪𝘻𝘰𝘯𝘵𝘢𝘭 𝘴𝘤𝘢𝘭𝘪𝘯𝘨 (𝘯𝘰𝘵 𝘷𝘦𝘳𝘵𝘪𝘤𝘢𝘭) 𝘧𝘪𝘹𝘦𝘴 𝘵𝘩𝘦 𝘣𝘰𝘵𝘵𝘭𝘦𝘯𝘦𝘤𝘬 𝘩𝘦𝘳𝘦. 💬 If you’re also fascinated by the architecture behind Databricks SQL Warehouses, I’d love to hear your take on this. #Databricks #DataEngineering #SQLWarehouse #Scalability #PerformanceOptimization #Lakehouse #DataArchitecture #SystemDesign

Databricks: View vs Temp View vs Materialized View — What’s the real difference? If you’ve worked with Databricks or Spark SQL, you’ve probably come across views, temp views, and materialized views. They might sound alike, but each serves a unique purpose 👇 🔹 View A saved SQL query stored in the catalog or metastore. Always pulls fresh data from the underlying tables. ✅ Ideal for reusable SQL logic and creating abstraction layers. 🔹 Temp View Exists only for the session (or cluster, if defined as global). Automatically disappears when the cluster or notebook stops. ✅ Perfect for quick experiments or intermediate transformations. 🔹 Materialized View Stores precomputed query results as a Delta table. Can be refreshed incrementally instead of recalculating everything. ✅ Best for performance-heavy queries and frequent aggregations. 👉 Rule of thumb: Use Views for governance and reusable logic. Use Temp Views for temporary, throwaway work. Use Materialized Views when performance is more important than up-to-the-second data freshness. hashtag#Databricks hashtag#DeltaLake hashtag#DataEngineering hashtag#ApacheSpark hashtag#DataPipelines hashtag#BigData hashtag#Analytics

🔗 SQL Joins Explained (Simply) Every data engineer uses joins daily — but many still get them mixed up 🤔 Here’s a quick guide to remember them 👇 🔹 INNER JOIN → Only matching rows 🔹 LEFT JOIN → All from left + matches 🔹 RIGHT JOIN → All from right + matches 🔹 FULL JOIN → Everything from both sides 💡 Joins are where SQL turns into real data engineering. 👉 Which JOIN do you use most often — INNER, LEFT, or FULL? 🔹 Check out my profile for more posts on Azure, ADF, Databricks, and Data Engineering concepts. 🔹 Follow / Connect for more Data Engineering insights, examples, and visuals. #DataEngineering #SQL #SQLServer #Database #Azure #AzureDataFactory #ETL #DataModeling #DataWarehouse #DataQuery #MicrosoftAzure #BigData #DataAnalytics #CloudData #LearningSQL

Excited to share a big milestone for Spark Notebooks: You can now query mirrored databases directly in your notebooks using SparkSQL or PySpark! Key Capabilities - Easily add MirrorDBs to your Spark notebook, just like Lakehouses. - Run read-only SparkSQL or PySpark queries on MirrorDB tables. Ensure your notebook either has no lakehouse attached or a schema-enabled lakehouse attached. - Run read-only queries without a Lakehouse by using fully qualified four-part names (workspace.database.schema.table) for precise querying. - Combine Mirrored DB tables with Lakehouse tables for unified insights across Fabric data assets. - Supported Mirrored DBs: Azure SQL Database, Cosmos DB, Snowflake, and Open Mirroring (with more coming soon). With this new capability, you can connect and run read-only queries on Mirrored DBs directly from your Spark notebooks, enabling seamless analytics across external data sources without migrating data into Fabric. Learn more: https://lnkd.in/gt63dkra #msfabric #spark #sparknotebooks

🚀 End-to-End ETL Architecture using Databricks & PySpark I built a modular ETL framework in Databricks that enables seamless data integration from multiple sources — including MySQL, MongoDB, S3, BigQuery, Oracle, and FTP — into diverse target systems such as Delta Tables, PostgreSQL, MySQL, and BigQuery. This framework allows you to: 🔹 Connect any data source using connectors or APIs 🔹 Write and customize business logic in Databricks notebooks (.ipynb) using PySpark 🔹 Load clean, transformed data into any target system (SQL, NoSQL, or cloud-based) 🔹 Handle dynamic column mappings and schema evolution efficiently 🧩 Tech Stack: Databricks | PySpark | MySQL | MongoDB | S3 | BigQuery | Delta | Unity Catalog 💡 Designed a flexible, plug-and-play ETL pipeline so users can focus on logic rather than configuration and infrastructure. 🔗 Check out the full code and implementation details on GitHub: 👉 https://lnkd.in/dGuDKWCC #DataEngineering #Databricks #ETL #PySpark #BigData #CloudData #DataPipeline #Azure #AWS #GCP

#StoredProcedures were demonised heavily in the latest years, but... still they live in the most stables systems out there. Now from the Data Engineering perspective, the market tried to cast out the stored procedures, in general, to give space to tools such as DBT (and name the people who write jinja sql stuff, Analytics Engineers), it was a similar same push when appeared the non-SQL hype. Just the time always prove who is correct and wrong. And today, now exists Stored Procedures even in Databricks, always existed in Snowflake, Google BigQuery, Oracle, PostgreSQL, etc. Never underestimate the power of the consolidated technologies, and the help they can do to solve problems.

Even though I write SQL from Databricks through DBeaver, it still feels like I’m bridging worlds, structured logic meets distributed systems. I haven’t touched the Databricks console yet(but, it's coming), even in this limited exposure, I’ve learned how environments shape how we think. Good data work doesn’t come from fancy tools; it comes from mastering fundamentals that carry across any platform. Whether it’s Oracle, SQL Server, or Databricks, clean logic wins every time. #SQL #DataEngineering