Skip to main content

Push description updates to Databricks Private Preview

TL;DR

Enable reverse sync to push asset descriptions authored in Atlan back to Databricks as comments.

Connect

Atlan can push asset descriptions written in Atlan back to Databricks as COMMENT values on the matching Unity Catalog objects. Your Databricks comments then stay in step with the descriptions your data stewards maintain in Atlan.

Description reverse sync is event driven. There is no workflow to schedule and no crawl to run. Once you enable it for a connection, Atlan picks up each description edit and applies it to Databricks for you.

Reverse sync for Databricks covers two kinds of metadata, and you enable each of them separately:

  • Descriptions - enabled per connection on the Databricks connector, as described on this page.
  • Tags - enabled per tag in the Governance center.

Enabling one of them has no effect on the other.

Did you know?

Description reverse sync is in private preview. To request access for your tenant, raise a support request.

Prerequisites​

Before you enable description reverse sync, make sure that you have the following:

  • A Unity Catalog-enabled workspace and a SQL warehouse configured.
  • Databricks privileges to update comments on your Databricks objects. The privileges differ by object type.
  • An existing Databricks connection in Atlan, with the assets you want to enrich already crawled.

Enable description reverse sync​

Who can do this?

Any admin or member user in Atlan can enable description reverse sync for a Databricks connection. You also need to work with your Databricks administrator to grant the privileges to update comments - you may not have access yourself.

Description Reverse Sync is a radio control with the values True and False, and it defaults to False. It appears in two places, and it does the same thing in both:

  • On the Databricks connector setup form, when you create a new connection.
  • On the workflow configuration form, when you modify the configuration of an existing Databricks workflow.

To enable description reverse sync:

  1. Open either the Databricks connector setup form or the configuration of an existing Databricks workflow.
  2. Locate the Description Reverse Sync control.
  3. Select True.
  4. Save your configuration.

The new setting takes a few minutes to take effect, and Atlan doesn't notify you when it does. Wait 5 to 7 minutes before you edit your first description, as described in What doesn't propagate. You don't need to run a crawl to enable description reverse sync, and you don't need to run one afterwards.

What propagates​

For each supported asset, Atlan runs the matching comment statement in Databricks:

Atlan assetStatement Atlan runs in Databricks
TableCOMMENT ON TABLE `catalog`.`schema`.`table` IS '...'
ViewCOMMENT ON VIEW `catalog`.`schema`.`view` IS '...'
SchemaCOMMENT ON SCHEMA `catalog`.`schema` IS '...'
ColumnALTER TABLE `catalog`.`schema`.`table` ALTER COLUMN `column` COMMENT '...'

A column description lands as a comment on that column, and a schema description lands as a comment on that schema. Each asset type keeps its own comment in Databricks.

What doesn't propagate​

  • Catalog descriptions. Atlan doesn't push catalog descriptions to Databricks, so a catalog description stays in Atlan.
  • Descriptions that Atlan imported from Databricks. Only a description that someone writes in Atlan triggers a push back to Databricks.
  • Edits you make while the control is set to False. Atlan discards those edits rather than queueing them.
  • Anything other than descriptions. Certificates, owners, README content, terms, and tags don't travel on this path. Tags have their own reverse sync.
  • View and schema descriptions, when you grant only MODIFY. A view comment needs ownership of the view, and a schema comment needs ownership of the schema or MANAGE on it. MODIFY covers tables and columns only. For the full list, see Grant permissions to push description updates.
Important!

Atlan discards any description edit you make while Description Reverse Sync is set to False. It doesn't queue those edits, and selecting True later doesn't back-fill them. To push an earlier edit to Databricks, edit the description again after you enable the control.

The same applies to the few minutes right after you select True, while the setting is still taking effect. Atlan doesn't notify you when the setting takes effect, so wait 5 to 7 minutes before you edit your first description. If you edited one during that window, edit it again.

What to expect​

  • A description you edit in Atlan reaches Databricks within about five minutes. Atlan checks for new edits every few minutes, so a single edit isn't instant.
  • To check the result in Databricks, open the asset in Catalog Explorer or run DESCRIBE on the asset and read its comment.

If a description doesn't reach Databricks:

  1. Confirm that Description Reverse Sync is set to True for that connection.
  2. Confirm your Databricks privileges for that object type. Views and schemas need ownership or MANAGE rather than MODIFY.
  3. Confirm that the asset is a table, view, schema, or column. Atlan doesn't push descriptions for any other asset type.
  4. Confirm that someone wrote the description in Atlan, rather than Atlan importing it from Databricks.

Now when your data stewards describe Databricks assets in Atlan, these description updates are also pushed to Databricks! 🎉

Did you know?

Description reverse sync only updates the comment on an existing Databricks object. It neither creates nor drops any object in Databricks.