Grab a spot →
Skip to main content

Crawl Dremio

Connect docs via MCP

Configure and run the crawler to extract metadata from your Dremio data lakehouse assets. After completing the prerequisite setup, you can crawl comprehensive metadata including physical datasets, virtual datasets, folders, spaces, and sources information. Review the order of operations for metadata enrichment workflows before starting.

Prerequisites​

Before you begin, make sure you have:

  • Completed Dremio setup
  • Admin access to your Atlan instance
  • Dremio connection details (host, port, credentials)

Create crawler workflow​

To crawl metadata from Dremio, review the order of operations and then complete the following steps.

  1. In your Atlan workspace, click Connectors in the left sidebar.

    • If you are using the Old UI (Classic), click New Workflow in the top navigation.

  2. Click Marketplace.

  3. Search for Dremio Assets and select it.

  4. Click Install.

  5. Once installation completes, click Setup Workflow on the same tile.

Configure authentication​

  1. Host: Enter your Dremio server hostname or IP address (without protocol or port)

  2. Port: Enter your Dremio REST API port (default: 443 for HTTPS)

  3. Authentication method, choose one:

    • Username/Password: Enter the username and password for the service account you created during setup.
    • Personal Access Token (PAT): Enter the PAT token (leave username empty)

  4. Click Test Connection to confirm connectivity to Dremio.

  5. Once successful, click Next.

Configure connection​

Complete the connection configuration for your Dremio environment:

  1. Provide a Connection Name that represents your source environment. For example, you might want to use values like production, development, gold, or analytics.

  2. To change the users able to manage this connection, change the users or groups listed under Connection Admins. If you don't specify any user or group, nobody can manage the connection—not even admins.

  3. To check for any permissions or configuration issues before running the crawler, click Preflight checks.

  4. At the bottom of the screen, click Next to proceed.

Advanced configuration​

This step is optional. Dremio is a query engine over external sources, so a physical dataset originates in another system that may already be cataloged in Atlan (a lakehouse catalog or database). To build lineage from a Dremio physical dataset back to that upstream source asset, expand Advanced Configuration and select the upstream Atlan connection for each source family:

  • Lakehouse Catalog Connections: the Atlan connections for Dremio's AWS Glue and Iceberg sources.
  • Database Connections: the Atlan connections for Dremio's relational sources (such as Snowflake, PostgreSQL, or Redshift).

Leave a family blank to skip it. The upstream source must already be crawled into Atlan for the lineage edge to resolve.

Run crawler​

After completing the configuration:

  • To run the crawler once, immediately, at the bottom of the screen click Run.
  • To schedule the crawler to run hourly, daily, weekly or monthly, at the bottom of the screen click Schedule & Run.

Verify crawled assets​

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

Atlan crawls comprehensive metadata from your Dremio data lakehouse, including physical datasets, virtual datasets, folders, spaces, and sources information.

  1. Monitor the crawler progress in the Workflows section:
    • View real-time status updates
    • Check the Logs tab for detailed execution information
    • Wait for the status to show Success before proceeding

See also​