Databricks assets package

The Databricks assets package crawls databricks assets and publishes them to Atlan for discovery.

Direct extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl assets directly from databricks:

Java

Coming soon

Python

Direct extraction from databricks

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import DatabricksCrawler

client = AtlanClient()

crawler = (
    DatabricksCrawler( # (1)
        client=client,  # (2)
        connection_name="production", # (3)
        admin_roles=[client.role_cache.get_id_for_name("$admin")],  # (4)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (5)
        allow_query=True, # (6)
        allow_query_preview=True, # (7)
    )
    .direct(hostname="test-hostname", port=443) # (8)
    .pat( # (9)
        access_token="test-pat",
        sql_warehouse_id="3d939b0cc668be06",
    )
    .metadata_extraction_method(
        type=DatabricksCrawler.ExtractionMethod.JDBC
    ) # (10)
    .include(assets={"test-include": ["ti1", "ti2"]}) # (11)
    .exclude(assets={"test-exclude": ["te1", "te2"]}) # (12)
    .exclude_regex(regex="TEST*")  # (13)
    .enable_view_lineage(False)  # (14)
    .enable_source_level_filtering(True)  # (15)
    .to_workflow() # (16)
)
response = client.workflow.run(crawler) # (17)

1. Base configuration for a new Databricks crawler.

2. You must provide a client instance.

3. You must provide a name for the connection that the Databricks assets will exist within.

4. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

5. You can specify a maximum number of rows that can be accessed for any asset in the connection. defaults to 10000.

6. You can specify whether you want to allow queries to this connection (default: True, as in this example) or deny all query access to the connection (False).

7. You can specify whether you want to allow data previews on this connection (default: True, as in this example) or deny all sample data previews to the connection (False).

8. When crawling assets directly from Databricks, you are required to provide the following information:

   • hostname of the Databricks instance.
   • port number of the Databricks instance. default: 443.

9. When using PAT authentication, you are required to provide the following information:

   • personal access token through which to access Databricks instance.
   • ID of the associated SQL warehouse eg: 3d939b0cc668be06

   You can also utilize any of the following authentication methods:

   • aws_service()

     • client_id: client ID for your AWS service principal.
     • client_secret: client secret for your AWS service principal.

   • azure_service()

     • client_id: client ID for Azure service principal.
     • client_secret: client secret for your Azure service principal.
     • tenant_id: tenant ID (directory ID) for Azure service principal.

10. Determines the interface that the package will use to extract metadata from Databricks.

    • REST: (recommended): use for Unity Catalog-enabled workspaces.
    • JDBC: use only if Unity Catalog is not available in your workspace.
    • SYSTEM_TABLES: use for unity catalog-enabled workspaces to extract metadata from system tables.

11. You can also optionally specify the set of assets to include in crawling. For Databricks assets, this should be specified as a map keyed by database name with values as a list of schemas within that database to crawl. (If set to [], all databases and schemas will be crawled.)

    tip

When using the DatabricksCrawler.ExtractionMethod.REST extraction method, ensure that you use the include_for_rest_api() method, which accepts a list of database names (ie: include_for_rest_api(assets=["te1", "te2"])) to include during crawling. ::: 12. You can also optionally specify the set of assets to exclude from crawling. For Databricks assets, this should be specified as a map keyed by database name with values as a list of schemas within the database to exclude. (If set to [], no assets will be excluded.).

tip

When using the DatabricksCrawler.ExtractionMethod.REST extraction method, ensure that you use the exclude_for_rest_api() method, which accepts a list of database names (ie: exclude_for_rest_api(assets=["te1", "te2"])) to exclude during crawling.

13. You can also optionally specify the exclude regex for the crawler to ignore collections based on a naming convention.

14. You can configure whether to enable view lineage as part of crawling Databricks (default: True).

15. You can configure advanced settings to enable (True) or disable (False) schema-level filtering on the source. Schemas specified in the include filter will be fetched.

16. Now, you can convert the package into a Workflow object.

17. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Extraction method: System tables

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl assets directly from databricks using system tables extraction method:

Java

Coming soon

Python

Direct extraction from databricks (system tables)

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import DatabricksCrawler

client = AtlanClient()

crawler = (
    DatabricksCrawler( # (1)
        client=client,  # (2)
        connection_name="production", # (3)
        admin_roles=[client.role_cache.get_id_for_name("$admin")],  # (4)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (5)
        allow_query=True, # (6)
        allow_query_preview=True, # (7)
    )
    .direct(hostname="test-hostname", port=443) # (8)
    .pat( # (9)
        access_token="test-pat",
        sql_warehouse_id="3d939b0cc668be06",
    )
    .metadata_extraction_method(
        type=DatabricksCrawler.ExtractionMethod.SYSTEM_TABLES
    ) # (10)
    .import_tags(include=False) # (11)
    .sql_warehouse(warehouse_ids=["3d939b0cc668be06"]) # (12)
    .enable_cross_workspace_discovery(include=False) # (13)
    .asset_selection_for_system_tables( # (14)
        selection_criteria=[
            DatabricksCrawler.AssetsSelection(
                type=DatabricksCrawler.AssetsSelectionCriteria.INCLUDE_BY_HIERARCHY,
                values={"db1": ["schema11", "schema12", "schema13"], "db2": [], "db3": ["schema31"]},
            ), # (15)
            DatabricksCrawler.AssetsSelection(
                type=DatabricksCrawler.AssetsSelectionCriteria.EXCLUDE_BY_HIERARCHY,
                values={"db4": ["schema41", "schema42", "schema43"]},
            ), # (16)
            DatabricksCrawler.AssetsSelection(
                type=DatabricksCrawler.AssetsSelectionCriteria.INCLUDE_BY_REGEX,
                values={
                    "asset_type": DatabricksCrawler.RegexAssetTypes.DATABASES,
                    "regex": ".*_INCLUDE_DB_REGEX",
                },
            ), # (17)
            DatabricksCrawler.AssetsSelection(
                type=DatabricksCrawler.AssetsSelectionCriteria.EXCLUDE_BY_REGEX,
                values={
                    "asset_type": DatabricksCrawler.RegexAssetTypes.SCHEMAS,
                    "regex": ".*_EXCLUDE_DB_REGEX",
                },
            ), # (18)
        ]
    )
    .enable_incremental_extraction(False) # (19)
    .to_workflow() # (20)
)
response = client.workflow.run(crawler) # (21)

1. Base configuration for a new Databricks crawler.

2. You must provide a client instance.

3. You must provide a name for the connection that the Databricks assets will exist within.

4. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

5. You can specify a maximum number of rows that can be accessed for any asset in the connection. defaults to 10000.

6. You can specify whether you want to allow queries to this connection (default: True, as in this example) or deny all query access to the connection (False).

7. You can specify whether you want to allow data previews on this connection (default: True, as in this example) or deny all sample data previews to the connection (False).

8. When crawling assets directly from Databricks, you are required to provide the following information:

   • hostname of the Databricks instance.
   • port number of the Databricks instance. default: 443.

9. When using PAT authentication, you are required to provide the following information:

   • personal access token through which to access Databricks instance.
   • ID of the associated SQL warehouse eg: 3d939b0cc668be06

   You can also utilize any of the following authentication methods:

   • aws_service()

     • client_id: client ID for your AWS service principal.
     • client_secret: client secret for your AWS service principal.

   • azure_service()

     • client_id: client ID for Azure service principal.
     • client_secret: client secret for your Azure service principal.
     • tenant_id: tenant ID (directory ID) for Azure service principal.

10. Determines the interface that the package will use to extract metadata from Databricks.

    • REST: (recommended): use for Unity Catalog-enabled workspaces.
    • JDBC: use only if Unity Catalog is not available in your workspace.
    • SYSTEM_TABLES: use for unity catalog-enabled workspaces to extract metadata from system tables.

11. You can configure whether to import tags from Databricks unity catalog to Atlan. Tags attached in Databricks will be automatically attached to your Databricks assets in Atlan.

12. Defines the filter for SQL warehouses to include when crawling eg: 3d939b0cc668be06

13. You can configure whether to enable cross-workspace discovery to discover assets from other workspaces. If True, cross-workspace discovery will be included while crawling Databricks, default: False.

14. Use asset_selection_for_system_tables to define the filter for system table assets to include / exclude when crawling. This method allows you to configure asset selection specifically for Databricks system tables using various selection criteria including hierarchical filtering and regex-based filtering.

    • selection_criteria: list of selection criteria objects containing the type of selection (include/exclude) and the corresponding values for filtering system table assets

15. You can also optionally specify the set of assets to include in crawling. For Databricks assets, this should be specified as a map keyed by database name with values as a list of schemas within that database to crawl. (If set to [], all databases and schemas will be crawled.)

    tip

When using the DatabricksCrawler.ExtractionMethod.REST extraction method, ensure that you use the include_for_rest_api() method, which accepts a list of database names (ie: include_for_rest_api(assets=["te1", "te2"])) to include during crawling. ::: 16. You can also optionally specify the set of assets to exclude from crawling. For Databricks assets, this should be specified as a map keyed by database name with values as a list of schemas within the database to exclude. (If set to [], no assets will be excluded.).

tip

When using the DatabricksCrawler.ExtractionMethod.REST extraction method, ensure that you use the exclude_for_rest_api() method, which accepts a list of database names (ie: exclude_for_rest_api(assets=["te1", "te2"])) to exclude during crawling.

17. You can also optionally specify the include regex (on different types: RegexAssetTypes.DATABASES/SCHEMAS/TABLE_VIEWS) for the crawler to include collections based on a naming convention.

18. You can also optionally specify the exclude regex (on different types: RegexAssetTypes.DATABASES/SCHEMAS/TABLE_VIEWS) for the crawler to ignore collections based on a naming convention.

19. You can configure whether to enable or disable schema incremental extraction on source. If True, incremental extraction will be included while crawling Databricks, default: False

20. Now, you can convert the package into a Workflow object.

21. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Offline extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl databricks assets from the S3 bucket:

Java

Coming soon

Python

Crawling databricks assets from a bucket

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import DatabricksCrawler

client = AtlanClient()

crawler = (
    DatabricksCrawler( # (1)
        client=client,  # (2)
        connection_name="production", # (3)
        admin_roles=[client.role_cache.get_id_for_name("$admin")],  # (4)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (5)
        allow_query=True, # (6)
        allow_query_preview=True, # (7)
    )
    .s3( # (8)
        bucket_name="test-bucket",
        bucket_prefix="test-prefix",
        bucket_region="test-region",
    )
    .to_workflow() # (9)
)
response = client.workflow.run(crawler) # (10)

1. Base configuration for a new Databricks crawler.

2. You must provide a client instance.

3. You must provide a name for the connection that the Databricks assets will exist within.

4. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

5. You can specify a maximum number of rows that can be accessed for any asset in the connection. defaults to 10000.

6. You can specify whether you want to allow queries to this connection (default: True, as in this example) or deny all query access to the connection (False).

7. You can specify whether you want to allow data previews on this connection (default: True, as in this example) or deny all sample data previews to the connection (False).

8. When using s3(), you need to provide the following information:

   • name of the bucket/storage that contains the extracted metadata files.
   • prefix is everything after the bucket/storage name, including the path.
   • (Optional) name of the region if applicable.

9. Now, you can convert the package into a Workflow object.

10. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Re-run existing workflow

To re-run an existing workflow for databricks assets:

Java

Coming soon

Python

Re-run existing databricks workflow

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.enums import WorkflowPackage

client = AtlanClient()

existing = client.workflow.find_by_type(  # (1)
  prefix=WorkflowPackage.DATABRICKS, max_results=5
)

# Determine which Databricks workflow (n)
# from the list of results you want to re-run.
response = client.workflow.rerun(existing[n]) # (2)

1. You can find workflows by their type using the workflow client find_by_type() method and providing the prefix for one of the packages. In this example, we do so for the DatabricksCrawler. (You can also specify the maximum number of resulting workflows you want to retrieve as results.)

2. Once you've found the workflow you want to re-run, you can simply call the workflow client rerun() method.

   • Optionally, you can use rerun(idempotent=True) to avoid re-running a workflow that is already in running or in a pending state. This will return details of the already running workflow if found, and by default, it is set to False.
   Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Requires multiple steps through the raw REST API

1. Find the existing workflow.
2. Send through the resulting re-run request.

POST /api/service/workflows/indexsearch

{
  "from": 0,
  "size": 5,
  "query": {
    "bool": {
      "filter": [
        {
          "nested": {
            "path": "metadata",
            "query": {
              "prefix": {
                "metadata.name.keyword": {
                  "value": "atlan-databricks" // (1)
                }
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "metadata.creationTimestamp": {
        "nested": {
          "path": "metadata"
        },
        "order": "desc"
      }
    }
  ],
  "track_total_hits": true
}

1. Searching by the atlan-databricks prefix will ensure you only find existing Databricks assets workflows.

   Name of the workflow

The name of the workflow will be nested within the _source.metadata.name property of the response object. (Remember since this is a search, there could be multiple results, so you may want to use the other details in each result to determine which workflow you really want.) :::

POST /api/service/workflows/submit

{
  "namespace": "default",
  "resourceKind": "WorkflowTemplate",
  "resourceName": "atlan-databricks-1684500411" // (1)
}

1. Send the name of the workflow as the resourceName to rerun it.