Skip to main content
Community Hub

Crawl Databricks Genie Agents

TL;DR

Discover and catalog Databricks Genie spaces as Genie Agent assets using Atlan's Databricks connector.

Your AI can read this via Docs MCPInstall MCP →Connect
Private Preview

Databricks Genie Agent support is in private preview and is enabled per tenant. The Import Genie Agents, Probe Genie Serialized Detail, and Enable Genie Agent Lineage options stay hidden in the workflow setup until the feature is enabled for your tenant. To request access, raise a support request.

Atlan can discover and catalog Databricks Genie spaces. A Genie space is a natural-language interface over a curated set of Databricks tables. Once crawled, each Genie space appears in Atlan as a Genie Agent asset alongside your other Databricks assets, and you can extract lineage from the Genie Agent to the Unity Catalog tables and columns it reads.

Genie metadata comes from the Databricks Genie REST API, not from Unity Catalog system tables. Genie Agent lineage comes from the system tables and runs in a separate workflow.

Genie Agents aren't Genie Code Skills

Import Genie Agents catalogs Genie spaces. It's a different option from Import Genie Code Skills, which catalogs SKILL.md files as Skill assets. Turning on one doesn't turn on the other.

Prerequisites​

Before crawling Genie Agents, make sure you have:

Permissions required​

In addition to the standard Databricks connector permissions, the Atlan service account must be able to read Genie spaces through the Databricks Genie API.

Probe Genie Serialized Detail needs more access than the rest of the crawl. It requires CAN EDIT on every Genie space you want the change token for. Without it, Atlan catalogs the space and leaves the change token empty rather than failing the crawl.

Configure crawler​

To configure the crawler for Genie Agents:

  1. Follow the standard Crawl Databricks steps.
  2. Turn on Import Genie Agents.
  3. Optional: turn on Probe Genie Serialized Detail to record each space's change token. This option appears only after you turn on Import Genie Agents.
  4. Run the workflow.

After the workflow completes, Genie Agent assets appear in Atlan under the crawled connection.

Both options default to off. When Import Genie Agents is off, the crawler makes no Genie API request and publishes no Genie Agent assets.

Turn off Genie Agent crawling​

If you turn off Import Genie Agents after a crawl has already published Genie Agents, the next run removes those assets from Atlan. The crawler stops extracting them, so they're absent from the snapshot Atlan compares against the previous run, and Atlan archives them.

Turning the option back on re-crawls your Genie spaces, and they reappear with their metadata on the next run.

Lineage isn't restored by the crawl alone

Re-enabling the option restores Genie Agent assets, but not their lineage. Run the lineage and usage workflow with Enable Genie Agent Lineage turned on to rebuild lineage.

What Atlan crawls​

For each Genie space, Atlan catalogs:

PropertyDescription
NameThe Genie space's title in Databricks
Space IDIdentifier of the Genie space in Databricks
Workspace IDIdentifier of the workspace containing the Genie space
Warehouse IDIdentifier of the SQL warehouse backing the Genie space
Parent pathWorkspace folder containing the Genie space. It's descriptive only and creates no hierarchy in Atlan
Change tokenSource change token for the Genie space. Populated only when Probe Genie Serialized Detail is on
DescriptionThe Genie space's description in Databricks, when it has one

Atlan doesn't read a Genie space's instructions, sample questions, example SQL, or verified answers, and doesn't store its serialized body.

See also​