Skip to main content

Monitor connectivity

Atlan runs its crawlers through an orchestrated set of automated tasks.

To monitor these orchestrated set of tasks follow

these steps.

Monitor the crawling process​

You can visualize the individual tasks a workflow runs as a directed acyclic graph (or "DAG").

To visualize the crawling process:

  • When running a workflow immediately, you will be redirected to the monitoring page within 5 seconds.
  • At any other moment:
    1. From the left menu, click Workflows to navigate to the Workflow center:
      • By default, workflow runs from the last 24 hours will be shown.
      • (Optional) Use the filters along the top to narrow down to the workflow run you want to monitor.
    2. From the Workflow run history table, click the workflow run you want to check.

On the left of the screen under the Summary tab, you can also see:

  • The current status of the workflow run.
  • The start and finish time of the workflow run.
  • The elapsed time (duration) of the workflow run.
  • Who triggered the workflow run and how (manually or automatically).

Identify errors​

If a crawler fails due to an error, Atlan will show where the failure occurred in the visualization.

To review the failure of any workflow with an error:

  1. Open the workflow run visualization (using either option above).
  2. Under the Summary tab on the left of the screen, click the View Failed tasks button.

Atlan will take you to the Failed Tasks tab on the left of the screen. Here you can review details about the specific activity or activities that failed.

Review log files​

Each task in the DAG may produce a log file containing additional details.

To review the log file for a specific activity:

  1. Click the task (activity node) in the DAG visualization.
  2. Open the Failed Tasks tab of a workflow run visualization (see steps above).
  3. To the right of each failed step, click the Logs button.

If there are any logs available, Atlan will display them on the screen.

Did you know?

Not every failed activity will produce a log. Look at the Message field of failed tasks for ideas about what went wrong when there is no log file available.

Manage all workflows​

You can monitor and manage all your workflows in Atlan from the workflow center.

To manage all your workflows:

  1. From the left menu of any screen in Atlan, click Workflows.
  2. From the tabs along the top in the Workflow center, click Manage.
  3. Search for a specific workflow from the search bar or click Select package to filter by supported connectors.
  4. (Optional) In the Filters menu on the left, select a filter to drill down further:
    • Click Created by to filter workflows created by specific users in Atlan.
    • Click Workflow type to filter workflows by type of workflow - connectors, utilities, and miners. Each workflow type also displays the total count of workflow runs for that type.
    • Click Schedule to filter workflows by scheduled or unscheduled runs.
  5. The workflow preview includes a summary of workflow details. Navigate to the workflow sidebar on the right, from the sidebar:
    • The Overview tab displays run count, when the workflow was created and by whom, workflow schedules if applicable, and last 5 runs. (Optional) Click Run workflow to run the workflow directly from the sidebar.
    • The Runs tab displays a summary of past workflow runs. (Optional) Select a workflow run to view more details or modify connectivity.
  6. Select any workflow to open the workflow.