Skip to main content

Crawl Google Cloud Dataplex

Configure and run the crawler to extract metadata from Google Cloud Dataplex. The crawler discovers Dataplex entries and their associated aspect metadata.

Prerequisites​

Before you begin, make sure you have:

Create crawler workflow​

To crawl metadata from Google Cloud Dataplex, review the order of operations and then complete the following steps.

  1. In the top right corner of any screen, click New and then click New Workflow.
  2. From the list of packages, select Google Dataplex and click Setup Workflow.

Configure authentication​

  1. Service Account JSON: Select the Google Cloud Service Account credential with Dataplex permissions that you created during setup.

  2. Location: Enter the Google Cloud location where your Dataplex aspect types are defined. The connector automatically expands the location to search for assets based on the following rules:

    • global searches for assets across all GCP locations
    • A multi-region such as us, eu, or asia searches all regions within that multi-region
    • A single region such as us-central1 searches only that specific region

    You can provide multiple locations separated by commas (for example, global or us, eu). The region- prefix is also supported (for example, region-global, region-us).

  3. Click Next to proceed to the configuration step.

Configure connection​

Complete the connection configuration for your Dataplex environment:

  1. Connection: (Optional) Select a BigQuery connection to filter entries. This limits the crawl to entries associated with the selected BigQuery connection. If not specified, all available entries are crawled.

  2. Include Aspect Types: (Optional) Select specific aspect types to include. If specified, only entries using these aspect types are extracted. Leave empty to extract all aspects.

    Note that aspect types can be used by entries in different locations. For example, an aspect type defined in global location can be used by tables in us location. Both locations must be provided in the Location field for complete discovery.

  3. Exclude Aspect Types: (Optional) Select aspect types to exclude. Entries using these aspects are skipped during extraction.

  4. Preflight Check: Click to run preflight checks and verify Google Dataplex API accessibility, authentication, and permissions before running the crawler.

  5. At the bottom of the screen, click Next to proceed.

Run crawler​

After completing the configuration:

  • To run the crawler once, immediately, at the bottom of the screen click Run.
  • To schedule the crawler to run hourly, daily, weekly or monthly, at the bottom of the screen click Schedule & Run.

Verify crawled assets​

Once the crawler has completed running, you can see the assets in Atlan's asset page.

  1. Monitor the crawler progress in the Workflows section:
    • View real-time status updates
    • Check the Logs tab for detailed execution information
    • Wait for the status to show Success before proceeding

See also​