## Crawl Starburst Enterprise

URL: https://docs.atlan.com/apps/connectors/database/starburst-enterprise/how-tos/crawl-starburst-enterprise

> Crawl metadata from Starburst Enterprise to catalog catalogs, schemas, tables, and data products. Extract domain definitions and lineage information after configuring user permissions.

Create a crawler workflow to automatically discover and catalog your [Starburst Enterprise](https://docs.atlan.com/llms/connectors/starburst-enterprise/set-up-starburst-enterprise/llms.txt) assets in Atlan. After completing the prerequisite setup, the crawler extracts metadata including catalogs, schemas, tables, views, materialized views, columns, domains, data products, and datasets. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Prerequisites

Before you begin, make sure you have:

* Configured [Starburst Enterprise user permissions](https://docs.atlan.com/llms/connectors/starburst-enterprise/set-up-starburst-enterprise/llms.txt)
* Created an [Atlan API token](https://docs.atlan.com/llms/connectors/starburst-enterprise/set-up-starburst-enterprise/llms.txt) with Admin privileges
* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for connection workflows
* Obtained your Starburst Enterprise host, port, and authentication credentials

## Create crawler workflow

1. In the top right of any screen, navigate to **New** and then click **New Workflow**.
2. From the list of packages, select **Starburst Enterprise** and click **Setup Workflow**.

### Configure extraction

Select your extraction method and provide the connection details.

### Direct

In **Direct** extraction, Atlan connects to your Starburst Enterprise instance and crawls metadata directly.

1. For **Host**, enter the hostname of your Starburst Enterprise coordinator.
2. For **Port**, enter the port (default `443`).
3. For **HTTP Scheme**, select **HTTPS** (recommended) or **HTTP**.
4. For **Authentication Method**, select either **Basic** or **LDAP**:
 * **Basic**: Enter the [username and password you created](https://docs.atlan.com/llms/connectors/starburst-enterprise/set-up-starburst-enterprise/llms.txt) for password file authentication.
 * **LDAP**: Enter your LDAP username and password.
5. For **Role**, enter the Trino role to use (default `sysadmin`). If you created a dedicated role for Atlan (for example, `atlan_metadata_reader`), enter that role name.
6. For **Atlan API Token**, enter the API token you [created during setup](https://docs.atlan.com/llms/connectors/starburst-enterprise/set-up-starburst-enterprise/llms.txt). The token must belong to a user with the **Admin** role.
7. For **Verify SSL**, keep **True** to validate TLS certificates or change to **False** for self-signed certificates.
8. Click **Test Authentication** to verify connectivity to your Starburst Enterprise instance.
9. Once successful, click **Next**.

### Agent

In **Agent** extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment. Use this when your Starburst Enterprise instance is behind a firewall and can't accept inbound connections from the internet.

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Select the **Agent** tab.
3. Store sensitive information (username, password) in the secret store configured with the Self-Deployed Runtime and reference the secrets in the corresponding fields. For more information, see [Configure secrets for workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).
4. For details on individual fields (host, port, HTTP scheme, role, Atlan API token, SSL verification), refer to the Direct extraction tab.
5. Click **Next** after completing the configuration.

### Configure connection

Set up your Starburst Enterprise connection name and define who can manage this connection in Atlan:

1. Provide a **Connection Name** that represents your source environment. For example, you might use values like `production`, `development`, or `analytics`.
2. To change the users able to manage this connection, change the users or groups listed under **Connection Admins**. If you don't specify any user or group, nobody can manage the connection, including admins.
3. Click **Next** to proceed.

### Configure crawler

Customize the crawler settings to control which assets are extracted from your Starburst Enterprise instance.

### Direct

1. For **Catalogs**, use the tree picker to select the catalogs and schemas you want to crawl. If no selection is made, all accessible catalogs are crawled.

2. For **Domains**, use the tree picker to select the domains and data products you want to crawl. If no selection is made, all accessible domains are crawled.

### Agent

When using Agent extraction, tree pickers aren't available. Instead, enter catalog and domain filters as JSON strings using **literal names** (not regex patterns).

1. For **Include Catalogs & Schemas**, enter a JSON object where keys are catalog names and values are arrays of schema names:

 ```json
 {"my_catalog": ["schema_a", "schema_b"], "other_catalog": ["public"]}
 ```

 To include all schemas in a catalog, use an empty array:

 ```json
 {"my_catalog": []}
 ```

 Leave the field empty or use `{}` to crawl all accessible catalogs.

2. For **Include Domains & Data Products**, enter a JSON object where keys are domain names and values are arrays of data product names:

 ```json
 {"Sales": ["Order Information", "Customer Data"]}
 ```

 Leave the field empty or use `{}` to crawl all accessible domains.

:::caution Use literal names, not regex patterns

Enter exact catalog, schema, domain, and data product names as they appear in Starburst Enterprise. Do **not** use regex anchors (for example, `^my_catalog$`). The filter values are passed directly to SQL queries and REST API calls as literal identifiers.

:::

3. For **Allow editing data products**, choose whether domains, data products, datasets, and dataset columns can be edited in Atlan:
 * **Yes** (default): Domains, data products, datasets, and dataset columns are editable in the Atlan UI.
 * **No**: Domains, data products, datasets, and dataset columns are published as read-only, preserving Starburst Enterprise as the source of truth.

4. For **Autoinclude catalog/schema**, keep **True** (default) to automatically include the backing catalog and schema when a data product is selected. This prevents gaps in metadata for selected data products.

:::info Autoinclude behavior

When a data product is selected and **Autoinclude catalog/schema** is enabled, the connector automatically includes the data product's backing catalog and schema in the extraction scope. This ensures you have both the data product assets and the underlying SQL assets for selected data products.

:::

### Run crawler

Execute the crawler workflow:

1. To check for any permissions or configuration issues before running the crawler, click **Preflight checks**. For details, see [Preflight checks for Starburst Enterprise](https://docs.atlan.com/llms/connectors/starburst-enterprise/preflight-checks-for-starburst-enterprise/llms.txt). Preflight checks are available for Direct extraction only.

2. You can either:
 * To run the crawler once immediately, click **Run**.
 * To schedule the crawler to run hourly, daily, weekly, or monthly, click **Schedule Run**.

Once the crawler has completed running, you can see the assets in Atlan's asset page.

## See also

* [How Atlan connects to Starburst Enterprise](https://docs.atlan.com/llms/connectors/starburst-enterprise/how-atlan-connects-to-starburst-enterprise/llms.txt): Understand connectivity options including Self-Deployed Runtime
* [What does Atlan crawl from Starburst Enterprise?](https://docs.atlan.com/llms/connectors/starburst-enterprise/what-does-atlan-crawl-from-starburst-enterprise/llms.txt): Learn about the metadata and assets that Atlan extracts from Starburst Enterprise
* [Data Product integration](https://docs.atlan.com/llms/connectors/starburst-enterprise/data-product-integration/llms.txt): Understand how SQL and data product assets are linked

---