## Crawl SageMaker

URL: https://docs.atlan.com/apps/connectors/lineage/amazon-sagemaker/how-tos/crawl-sagemaker

> Crawl lineage from AWS SageMaker to catalog machine learning pipelines, jobs, models, and datasets. Extract ML workflow lineage after configuring AWS credentials.

Configure and run the [SageMaker](https://docs.atlan.com/llms/connectors/amazon-sagemaker/set-up-sagemaker/llms.txt) crawler to extract lineage from your machine learning workflows and catalog your ML assets in Atlan. After completing the prerequisite setup, you can extract lineage information from SageMaker pipelines, training jobs, models, and datasets. Review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for metadata enrichment workflows before starting.

## Prerequisites

Before you begin, make sure you have:
* Completed [SageMaker setup](https://docs.atlan.com/llms/connectors/amazon-sagemaker/set-up-sagemaker/llms.txt)
* Admin or connection admin privileges in Atlan
* AWS credentials (Access Key ID and Secret Access Key or IAM Role ARN)
* AWS region where your SageMaker resources are located

## Create crawler workflow

Follow these steps to create a workflow in Atlan that captures metadata from SageMaker.

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **SageMaker** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure authentication

Configure authentication for your extraction method:

* In **Direct** extraction, Atlan connects to your AWS SageMaker service and crawls metadata directly.
* In **Agent** extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment.

### Direct extraction

In **Direct** extraction, Atlan connects to your AWS SageMaker service and crawls metadata directly.

1. **Extraction method**: Select **Direct**

2. Choose your authentication method:
 - **IAM User**: Enter your AWS Access Key ID and Secret Access Key
 - **IAM Role**: Enter your IAM Role ARN for cross-account access
3. Enter your AWS credentials:
	* **AWS Region**: Enter your primary SageMaker region (for example, `us-east-1`)
	* For **IAM User**: 
	 - **AWS Access Key ID**: Enter your AWS Access Key ID
	 - **AWS Secret Access Key**: Enter your AWS Secret Access Key
	* For **IAM Role**:
	 - **AWS Role ARN**: Enter your IAM Role ARN for cross-account access
	 - (Optional) **External ID**: Enter the external ID provided by Atlan support

4. Click **Test Connection** to verify your AWS credentials work correctly.

5. Once successful, click **Next**.

### Agent extraction

Use Agent extraction when your AWS account or SageMaker service isn't reachable from Atlan Cloud (for example, it's behind a firewall). A Self-Deployed Runtime runs inside your network and connects to your AWS SageMaker service, then sends metadata to Atlan over an outbound connection.

Before configuring the crawler:

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Confirm the runtime can reach your AWS SageMaker service over your local network and that [network security](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-network-security/llms.txt) is configured.

To configure the crawler:

1. **Extraction method**: Select **Agent**

2. Configure AWS credentials by adding the secret keys for your secret store. For details on the required fields, refer to the [Direct extraction](#configure-authentication) section.

3. Complete the Secure Agent configuration by following the instructions in the [How to configure Secure Agent for workflow execution](https://docs.atlan.com/llms/platform/secure-agent/configure-secure-agent-for-workflow-execution/llms.txt) guide.

4. Click **Next** after completing the configuration.

### Configure connection

To complete the Sagemaker connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`.

2. (Optional) To change the users able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning

 If you don't specify any user or group, nobody can manage the connection - not even admins.

 :::

3. At the bottom of the screen, click **Next** to proceed.

### Run crawler

To run the Sagemaker crawler, after completing the previous steps:

1. To check for any permissions or other configuration issues before running the crawler, click **Preflight checks**.

2. You can either:
 - To run the crawler once immediately, at the bottom of the screen, click the **Run** button.
 - To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule Run** button.

Once the crawler has completed running, you can see the assets in Atlan's asset page! 🎉

## Troubleshooting

If you encounter connection or authentication issues during the crawl setup, see [Connection and authentication issues](https://docs.atlan.com/llms/connectors/amazon-sagemaker/connection-and-authentication-issues/llms.txt) for detailed troubleshooting steps.

## See also

* [What does Atlan crawl from SageMaker](https://docs.atlan.com/llms/connectors/amazon-sagemaker/what-does-atlan-crawl-from-sagemaker/llms.txt): Learn what assets and metadata Atlan extracts from SageMaker

---