We’re excited to invite you to our MONTHLY Databricks Get Started Days, a half-day virtual learning experience designed to jumpstart your journey with Databricks. This global event is tailored to equip you with essential data engineering and analysi...
- 343 Views
- 0 replies
- 0 kudos
Thursday
AI is transforming BI by changing the way organizations manage, analyze and get insights from their data. Everyone is rushing to democratize access to data through LLMs. But without the right foundation, you can’t get the most from AI. Our new appro...
- 215 Views
- 0 replies
- 0 kudos
Thursday
Join us for the return of the Virtual Learning Festival! Mark your calendars from 9 April - 30 April 2025! Upskill across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career with...
- 18886 Views
- 55 replies
- 17 kudos
03-03-2025
Date & Time:8 April (Tuesday), 11 AM – 3 PM SGT Registration Page:Register Here: https://events.databricks.com/FY260408-SD-AIBIforSelf-ServiceAnalytics What You'll Learn:In this training session, you will learn how to self-serve business insights ...
- 2565 Views
- 1 replies
- 2 kudos
2 weeks ago
Learn the foundational principles of the lakehouse architecture with three short self-paced videos The architecture of your data platform directly affects the reliability, performance and scalability of your data and AI initiatives. The lakehouse arc...
- 5240 Views
- 0 replies
- 2 kudos
a month ago
Be part of a global movement! Connect with 22,000 data enthusiasts at 700+ sessions, keynotes, and hands-on training at this year’s Data + AI Summit. Whether you’re into data intelligence, governance, AI, or data warehousing, this is your chance to l...
- 6081 Views
- 1 replies
- 3 kudos
02-19-2025
Meet Takuya Omi (尾美拓哉), a passionate Data Engineer at NS Solutions and a valuable member of the Databricks Community. Takuya's career journey began in the marketing division of a financial institution, where his early work with SQL and machine learni...
- 2949 Views
- 4 replies
- 6 kudos
Monday
Community Activity
- 148 Views
- 2 replies
- 1 kudos
Optimizing Spark Read Performance on Delta Tables with Deletion Vectors Enabled
Hi Databricks Experts,I'm currently using Delta Live Table to generate master data managed within Unity Catalog, with the data stored directly in Google Cloud Storage. I then utilize Spark to read these master data from the GCS bucket. However, I’m ...
- 148 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi Hung, The performance issues you're experiencing with Delta tables and deletion vectors are common challenges when working with Delta Live Tables. Let me address your questions: How to Optimize Spark Read Performance with Deletion Vectors Deletion...
- 1 kudos
- 87 Views
- 1 replies
- 0 kudos
Urgent Issue - Databricks Data Engineer Professional Exam Suspended
Hello Databricks Community - @Cert-TeamOPS ,I am writing with significant concern and frustration regarding my Databricks Data Engineer Professional exam that was scheduled for today, April 4, 2025, at 3:30 PM EDT.I started the exam on time, having p...
- 87 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@Cert-Team , @Cert-TeamOPS , @DB_Paul - adding for attention and help
- 0 kudos
- 197 Views
- 2 replies
- 0 kudos
Build a streaming table on top of a snowflake table
Is it possible to create a streaming table on top of a snowflake tablel accessible via the Lakehouse Federation?
- 197 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi abelian-grape,How are you doing today?, As per my understanding, Right now, it’s not possible to create a streaming table directly on top of a Snowflake table that’s accessed through Lakehouse Federation in Databricks. Lakehouse Federation allows ...
- 0 kudos
- 293 Views
- 2 replies
- 0 kudos
Triggering Downstream Workflow in Databricks from New Inserts in Snowflake
Hi Databricks experts,I have a table in Snowflake that tracks newly added items, and a downstream data processing workflow that needs to be triggered whenever new items are added. I'm currently using Lakehouse Federation to query the Snowflake tables...
- 293 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi abelian-grape,Great question! Since you're using Lakehouse Federation to access the Snowflake table, and Databricks can't directly stream from or listen to inserts in Snowflake, the best approach is to use an interval-based polling mechanism in Da...
- 0 kudos
- 36769 Views
- 142 replies
- 9 kudos
Suspended Exam? Don't worry! Our support team can help!
Taking exams is so stressful already, and if your exam gets suspended it is extra stressful. Don't worry! Our support team is awesome and will help you get back on track. The most important step is to file a ticket in our Help Center. Be sure to inc...
- 36769 Views
- 142 replies
- 9 kudos
Latest Reply
- 9 kudos
@Cert-Team I had my certification exam for Databricks Data Engineer Professional scheduled today 3:30PM ET. As soon as I started the exam, within 5 minutes I was told to pause and show the room around; and it was clear per the exam policy. Immediatel...
- 9 kudos
- 698 Views
- 1 replies
- 1 kudos
Vnet Gateway issues on Power bi Conn
Team,We are getting frequent vnet gateway failures on power bi Dataset using DAX(simple DAX not complex) and upon the rerun it is working, is any permanent fix for this ?Error :{"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error...
- 698 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Similar issue. Any solution?
- 1 kudos
- 57 Views
- 0 replies
- 0 kudos
Databricks Access Management
Hi, Can someone suggest me what's the least level of access we can provide to any external business user to able to run a notebook and download the results. I tried with giving read and run level permissions at notebook level and also with a user gro...
- 57 Views
- 0 replies
- 0 kudos
- 231 Views
- 3 replies
- 2 kudos
Who’s heading to the Data & AI Summit in San Francisco this year?
The Data + AI Summit is happening June 9-12, and we are gearing up for an incredible in-person experience! With 22,000+ data enthusiasts, 700+ sessions, and hands-on training, this is where innovation meets collaboration. Whether you are a data leade...
- 231 Views
- 3 replies
- 2 kudos
- 15867 Views
- 18 replies
- 18 kudos
Resolved! How to connect Neo4j aura to a cluster
Please help resolve this error :org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the serverThis occurs when I want to establish a connection to neo4j aura to my cluster .Thank you.
- 15867 Views
- 18 replies
- 18 kudos
Latest Reply
- 18 kudos
I have added the init script on cluster start up in Spark->config->init.scripts. But my clsuter won't start after this.Cluster scoped init script /Volumes/xxx/xxx/neo4j/neo4j-init.sh failed: Script exit status is non-zero. Could you please help me or...
- 18 kudos
- 157 Views
- 1 replies
- 0 kudos
.py file running stuck on waiting
Hello, hope you are doing well.We are facing an issue when running .py files. This is fairly recent and we were not experiencing this issue last week.As shown in the screenshots below, the .py file hangs on "waiting" after we press "run all". No matt...
- 157 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey @BigAlThePal, It looks like a UI bug, especially in Microsoft Edge. The code actually runs, but the output doesn't show until you refresh. A few quick things you can try:Run cells individually instead of using "Run All"Switch to Chrome or Firefox...
- 0 kudos
- 674 Views
- 2 replies
- 1 kudos
Getting error while creating external delta table in Databricks
I am getting below error while creating external delta table in Databricks, even there is a external location created.[NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] No parent external location was found for path 'abfss://destination@datalakeprojectsid.dfs.co...
- 674 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @Siddalinga,Is the issue fixed ? If not please check below options : 1. external location is created in databricks with proper service account or service principle ? 2. Or any storage location is attached with proper service account. ?Above 2 m...
- 1 kudos
- 353 Views
- 2 replies
- 0 kudos
I’m having trouble connecting to my AWS S3 bucket
Hello,I’m having trouble connecting to my AWS S3 bucket from my local Python environment using boto3. When I try to run the following code:import boto3s3 = boto3.client('s3')response = s3.list_objects_v2(Bucket='my-bucket-name')print(response)error m...
- 353 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Are you trying from databricks notebook? if so can you share the dbr version ?If not, you can install the below code :import sslprint(ssl.OPENSSL_VERSION)and update the pip package : `pip install --upgrade pip` try post this. Happy learnin...
- 0 kudos
- 593 Views
- 2 replies
- 0 kudos
Data Management and Governance with Unity Catalog Demo Set up error
I am trying to run classroom setup for Demo:Upgrading Tables to Unity Catalog lab in the above course and get multiple errors. I wasn able to complete previous 2 labs successfully.Here are some errors while running the classrom set up script "%run ./...
- 593 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Only few courses are allowing to change the table metadata or type itself for the labsessions. In this case you can download and try in your workspace(if any) that will work easily for the learning purpose.Thanks for the ask, happy learnin...
- 0 kudos
- 702 Views
- 1 replies
- 0 kudos
set up my AWS credentials and configure boto3, but I’m facing an issue
Hi everyone,I’m working on a project where I need to upload files from my local machine to an AWS S3 bucket using the boto3 library in Python. I’ve followed the official documentation to set up my AWS credentials and configure boto3, but I’m facing a...
- 702 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Thanks for the ask, are you trying to read via notebook ? If so you need to pass the credentials either cluster level or best approach via secret scopes then it will work.Please ignore if not matching with your ask.Saran
- 0 kudos
- 581 Views
- 1 replies
- 0 kudos
How to check databricks-connect types of objects.
While using `databricks-sdk` in my code, I've found that checking PySpark objects types is not reliable anymore. I've used to do the following: from pyspark.sql import Column, DataFrame, SparkSession isinstance(spark, SparkSession) isinstance(a_df...
- 581 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello Mate,Is the issue fixed ? did you got any reference or support from databricks end ?Saran
- 0 kudos
Announcements
Latest from our Blog
The Ultimate Guide to Chunking Strategies for RAG Applications with Databricks
Over the years, I have collaborated closely with ML engineering leaders across various industries, guiding them on how to make the right chunking strategy decisions for their Retrieval-Augmented Gener...
Migrating ML Workload to Databricks
As machine learning (ML) workloads continue to grow in complexity and scale, organizations are looking for efficient and scalable solutions to manage their ML lifecycle. Databricks offers a powerful p...
Building Trust in Data II: A Guide to Effective Data Testing Tactics
Introduction Building a reliable data pipeline goes beyond setting up a functional workflow — it requires meticulous testing to ensure data accuracy, integrity, and quality across every stage of the p...
Building Trust in Data I: A Guide to Effective Data Testing Frameworks
Introduction When should I test my data? How should it be done? Which tools and testing methods are most effective? Who holds responsibility for data quality?” These are common questions that arise in...
