Skip to main content

Looker assets package

The Looker assets package crawls Looker assets and publishes them to Atlan for discovery.

Direct extraction​

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl Looker assets directly from Looker:

Direct extraction from Looker
Workflow looker = LookerCrawler.directResourceOwner( // (1)
        "production", // (2)
        "https://example.cloud.looker.com", // (3)
        443, // (4)
        "pEBDcOauyVfCtQacdvJUL", // (5)
        "eKnUIFK2JvmVO5mruqbxjnzXf", // (6)
        "-----BEGIN OPENSSH PRIVATE KEY-----\n-99+PmSlex0FmY9ov1J8H1H9Y3IMWXbL...\n-----END OPENSSH PRIVATE KEY-----", // (7)
        "b3WnMIjaZJaTYZd", // (8)
        List.of(client.getRoleCache().getIdForName("$admin")), // (9)
        null,
        null,
        List.of("67"), // (10)
        List.of("dbt_food_beverage"), // (11)
        null, // (12)
        null); // (13)
WorkflowResponse response = looker.run(client); // (14)

  1. The LookerCrawler package will create a workflow to crawl assets from Looker. The directResourceOwner() method creates a workflow for crawling assets directly from Looker.

  2. You must provide a name for the connection that the Looker assets will exist within.

  3. You must provide the hostname of your Looker instance.

  4. You must specify the port number of the Looker instance (use 443 for the default).

  5. You must provide your admin client ID.

  6. You must provide your admin client secret.

  7. If you want to crawl field-level lineage, you must provide your SSH private key.

  8. If you want to crawl field-level lineage, you must provide the passphrase for your SSH private key (if any).

  9. You must specify at least one connection admin, either:

    • everyone in a role (in this example, all $admin users)
    • a list of groups (names) that will be connection admins
    • a list of users (names) that will be connection admins

  10. You can also optionally specify the list of folders to include in crawling. For Looker assets, this should be specified as a list of numeric folder IDs. (If set to null, all folders will be crawled.)

  11. You can also optionally specify the list of projects to include in crawling. For Looker assets, this should be specified as a list of project names. (If set to null, all projects will be crawled.)

  12. You can also optionally specify the list of folders to exclude from crawling. For Looker assets, this should be specified as a list of numeric folder IDs. (If set to null, no folders will be excluded.)

  13. You can also optionally specify the list of projects to exclude from crawling. For Looker assets, this should be specified as a list of project names. (If set to null, no projects will be excluded.)

  14. You can then run the workflow using the run() method on the object you've created. Because this operation will execute work in Atlan, you must provide it an AtlanClient through which to connect to the tenant.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Re-run existing workflow​

To re-run an existing workflow for Looker assets:

Re-run existing Looker workflow
List<WorkflowSearchResult> existing = WorkflowSearchRequest // (1)
    .findByType(client, LookerCrawler.PREFIX, 5); // (2)
// Determine which of the results is the Looker workflow you want to re-run...
WorkflowRunResponse response = existing.get(n).rerun(client); // (3)

  1. You can search for existing workflows through the WorkflowSearchRequest class.

  2. You can find workflows by their type using the findByType() helper method and providing the prefix for one of the packages. In this example, we do so for the LookerCrawler. (You can also specify the maximum number of resulting workflows you want to retrieve as results.)

  3. Once you've found the workflow you want to re-run, you can simply call the rerun() helper method on the workflow search result. The WorkflowRunResponse is just a subtype of WorkflowResponse so has the same helper method to monitor progress of the workflow run. Because this operation will execute work in Atlan, you must provide it an AtlanClient through which to connect to the tenant.

    • Optionally, you can use the rerun(true) method with idempotency to avoid re-running a workflow that is already in running or in a pending state. This will return details of the already running workflow if found, and by default, it is set to false
    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Was this page helpful?