## Crawl Dremio

URL: https://docs.atlan.com/apps/connectors/data-lakehouses/dremio/how-tos/crawl-dremio

> Crawl metadata from Dremio to catalog physical datasets, virtual datasets, folders, and spaces. Extract lineage and asset information from your data lakehouse.

Configure and run the crawler to extract metadata from your [Dremio](https://docs.atlan.com/llms/connectors/dremio/set-up-dremio/llms.txt) data lakehouse assets. After completing the prerequisite setup, you can crawl comprehensive metadata including physical datasets, virtual datasets, folders, spaces, and sources information. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Prerequisites

Before you begin, make sure you have:
* Completed [Dremio setup](https://docs.atlan.com/llms/connectors/dremio/set-up-dremio/llms.txt)
* Admin access to your Atlan instance
* Dremio connection details (host, port, credentials)

## Create crawler workflow

To crawl metadata from Dremio, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps.

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Dremio Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure authentication

1. **Host**: Enter your Dremio server hostname or IP address (without protocol or port)

2. **Port**: Enter your Dremio REST API port (default: `443` for HTTPS)

3. **Authentication method**, choose one:
 * **Username/Password**: Enter the username and password for the service account you created during setup.
 * **Personal Access Token (PAT)**: Enter the PAT token (leave username empty)
4. Click **Test Connection** to confirm connectivity to Dremio.

5. Once successful, click **Next**.

### Configure connection

Complete the connection configuration for your Dremio environment:

1. Provide a **Connection Name** that represents your source environment. For example, you might want to use values like `production`, `development`, `gold`, or `analytics`.

2. To change the users able to manage this connection, change the users or groups listed under **Connection Admins**. If you don't specify any user or group, nobody can manage the connection—not even admins.

3. To check for any permissions or configuration issues before running the crawler, click **Preflight checks**.

4. At the bottom of the screen, click **Next** to proceed.

### Advanced configuration

This step is optional. Dremio is a query engine over external sources, so a physical dataset originates in another system that may already be cataloged in Atlan (a lakehouse catalog or database). To build lineage from a Dremio physical dataset back to that upstream source asset, expand **Advanced Configuration** and select the upstream Atlan connection for each source family:

- **Lakehouse Catalog Connections**: the Atlan connections for Dremio's AWS Glue and Iceberg sources.
- **Database Connections**: the Atlan connections for Dremio's relational sources (such as Snowflake, PostgreSQL, or Redshift).

Leave a family blank to skip it. The upstream source **must already be crawled into Atlan** for the lineage edge to resolve.

## Run crawler

After completing the configuration:

* To run the crawler once, immediately, at the bottom of the screen click **Run**.
* To schedule the crawler to run hourly, daily, weekly or monthly, at the bottom of the screen click **Schedule & Run**.

### Verify crawled assets

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

Atlan crawls comprehensive metadata from your Dremio data lakehouse, including physical datasets, virtual datasets, folders, spaces, and sources information.

1. Monitor the crawler progress in the **Workflows** section:
 * View real-time status updates
 * Check the **Logs** tab for detailed execution information
 * Wait for the status to show **Success** before proceeding

## See also

* [What does Atlan crawl from Dremio](https://docs.atlan.com/llms/connectors/dremio/what-does-atlan-crawl-from-dremio/llms.txt): Learn what assets, metadata, and lineage Atlan crawls from Dremio

---