Atlan natively supports Databricks, which allows you to seamlessly integrate your tables from Databricks with your Atlan workspace.
You can set up a Databricks integration with your Atlan workspace in four easy steps:
Select the source, aka Databricks 😉
Provide your credentials ✍️
Set up your configuration 🗄️
Schedule automatic updates 🕑
Before you start integrating your Databricks cluster with Atlan, you'll need some information that will help establish a connection between Atlan and your Databricks Account:
Go to the Databricks console and select "Clusters" from the left sidebar.
Select the cluster you want to connect with Atlan. The cluster should be in a
Running state for the Atlan crawler to fetch metadata from it.
Click on "Advanced Options" in the "Configuration" tab.
Select the "JDBC/ODBC" tab and copy the information here:
Host: Databricks cluster server hostname
Port: Port. Typically it is
Personal Access Token: You can generate the Personal Access Token by following the official guide.
JDBC URL Suffix: This is the JDBC URL suffix. Highlighted in the image above. Please ensure you do no add the hostname, port or PWD values here.
Once you have the prerequisite information listed above, please follow the steps below to establish a connection and integrate Atlan with your Databricks cluster.
Log into your Atlan workspace.
On the home screen, click on the "New Integration" button in the top right corner. You will see a dialogue box with the list of sources available on your workspace.
Select "Databricks" from the list of options, and click on "Next".
You will see an option to either select a preconfigured credential from the drop-down menu or to create a credential. To set up a new connection, click on the "Create Credential" button.
You will be required to fill in your Databricks credentials.
Once you have filled in the details, click on "Next".
You will now be asked to fill in the details of your database and table.
Chose whether to run the crawler once or schedule it for a daily, weekly, or monthly run. You will be asked to specify the timezone for the run.
Click on "Create". Your connection is now created.
Congratulations! You have now integrated Atlan with your Databricks cluster 🎉
Once the integration setup is completed, you will be redirected to the "Monitor" tab for your Databricks asset, where you can monitor its progress.