Steps to integrate your Databricks cluster with Atlan

Atlan natively supports Databricks, which allows you to seamlessly integrate your tables from Databricks with your Atlan workspace.

๐Ÿ’ญ TL;DR

You can set up a Databricks integration with your Atlan workspace in four easy steps:

  1. Select the source, aka Databricks ๐Ÿ˜‰

  2. Provide your credentials โœ๏ธ

  3. Set up your configuration ๐Ÿ—„๏ธ

  4. Schedule automatic updates ๐Ÿ•‘

๐Ÿ“œ Prerequisites for Databricks integration

Before you start integrating your Databricks cluster with Atlan, you'll need some information that will help establish a connection between Atlan and your Databricks Account:

  1. Go to the Databricks console and select "Clusters" from the left sidebar.

  2. Select the cluster you want to connect with Atlan. The cluster should be in a Running state for the Atlan crawler to fetch metadata from it.

  3. Click on "Advanced Options" in the "Configuration" tab.

  4. Select the "JDBC/ODBC" tab and copy the information here:

Databricks cluster configuration
  • Host: Databricks cluster server hostname

  • Port: Port. Typically it is 443.

  • Personal Access Token: You can generate the Personal Access Token by following the official guide.

  • JDBC URL Suffix: This is the JDBC URL suffix. Highlighted in the image above. Please ensure you do no add the hostname, port or PWD values here.

๐ŸŒŸ Pro Tip: If you don't have this information handy, reach out to your cloud or data lake administrator to get these details before you get started!

๐Ÿš€ The step-by-step guide to integrating Databricks with Atlan

Once you have the prerequisite information listed above, please follow the steps below to establish a connection and integrate Atlan with your Databricks cluster.

STEP 1: Selecting the source

  1. Log into your Atlan workspace.

  2. On the home screen, click on the "New Integration" button in the top right corner. You will see a dialogue box with the list of sources available on your workspace.

  3. Select "Databricks" from the list of options, and click on "Next".

STEP 2: Providing credentials

  1. You will see an option to either select a preconfigured credential from the drop-down menu or to create a credential. To set up a new connection, click on the "Create Credential" button.

  2. You will be required to fill in your Databricks credentials.

  3. Once you have filled in the details, click on "Next".

Databricks credential configuration

STEP 3: Setting up your configuration

  1. You will now be asked to fill in the details of your database and table.

  2. Chose whether to run the crawler once or schedule it for a daily, weekly, or monthly run. You will be asked to specify the timezone for the run.

  3. Click on "Create". Your connection is now created.

Congratulations! You have now integrated Atlan with your Databricks cluster ๐ŸŽ‰

๐Ÿ Monitoring your Databricks Integration

Once the integration setup is completed, you will be redirected to the "Monitor" tab for your Databricks asset, where you can monitor its progress.