## Crawl Amazon MSK

URL: https://docs.atlan.com/apps/connectors/messaging/amazon-msk/how-tos/crawl-amazon-msk

> To crawl metadata from Amazon MSK, review the [order of operations](/product/connections/how-tos/order-workflows) and then complete the following steps.

Once you have [configured the Amazon MSK permissions](https://docs.atlan.com/llms/connectors/amazon-msk/set-up-amazon-msk/llms.txt), you can establish a connection between Atlan and Amazon MSK. (If you are also using a private network for Amazon MSK, you need to [set that up first](https://docs.atlan.com/llms/connectors/amazon-msk/set-up-a-private-network-link-to-amazon-msk/llms.txt), too.)

To crawl metadata from Amazon MSK, review the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) and then complete the following steps.

## Select the source

To select Amazon MSK as your source:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Amazon MSK Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

## Provide credentials

To enter your Amazon MSK credentials:

1. For _Extraction method_, _Direct_ is the default selection.

2. For _Bootstrap servers_, enter the hostname(s) (or [PrivateLink cluster connection string](https://docs.atlan.com/llms/connectors/amazon-msk/set-up-a-private-network-link-to-amazon-msk/llms.txt)) of your Amazon MSK broker(s) - for multiple hostnames, separate each entry with a comma `,` or semicolon `;`.

3. For _Authentication_, _IAM Role_ is the default authentication method.

4. For _Deployment Type_, _Provisioned_ is the default deployment type.

5. For _Security protocol_, _SASL\_SSL_ is the default security protocol.

6. For _AWS Role ARN_, enter the ARN of the [IAM role you created in your AWS account](https://docs.atlan.com/llms/connectors/amazon-msk/set-up-amazon-msk/llms.txt).

7. For _AWS Region_, enter the AWS region of your Amazon MSK cluster.

8. Click the **Test Authentication** button to confirm connectivity to Amazon MSK.

9. Once authentication is successful, navigate to the bottom of the screen and click **Next**.

## Configure the connection

To complete the Amazon MSK connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, you might use values like `production`,`development`,`gold`, or `analytics`.

2. (Optional) To change the users who are able to manage this connection, change the users or groups listed under _Connection Admins_.

 :::warning

 If you do not specify any user or group, no one will be able to manage the connection - not even admins.

 :::

3. Navigate to the bottom of the screen and click **Next** to proceed.

## Configure the crawler

Before running the Amazon MSK crawler, you can further configure it.

On the _Metadata_ page, you can override the defaults for any of these options:

- For _Skip internal topics_, keep the default option **Yes** to skip [internal Apache Kafka topics](https://kafka.apache.org/11/documentation/streams/developer-guide/manage-topics#streams-developer-guide-topics-internal) or click **No** to enable crawling them.
- To have the crawler ignore topics based on a naming convention, specify a regular expression in the _Exclude topic regex_ field.
- To have the crawler include topics based on a naming convention, specify a regular expression in the _Include topic regex_ field.

:::info **Did you know?** 
 If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

:::

## Run the crawler

To run the Amazon MSK crawler, after completing the steps above:

1. To check for any [permissions or other configuration issues](https://docs.atlan.com/llms/connectors/amazon-msk/preflight-checks-for-amazon-msk/llms.txt) before running the crawler, click **Preflight checks**.

2. You can either:
 - To run the crawler once immediately, at the bottom of the screen, click the **Run** button.
 - To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule & Run** button.

Once the crawler has completed running, you will see the assets on Atlan's asset page! 🎉

---