## Crawl Microsoft Azure Cosmos DB for NoSQL

URL: https://docs.atlan.com/apps/connectors/database/microsoft-azure-cosmos-db-for-nosql/how-tos/crawl-microsoft-azure-cosmos-db-for-nosql

> Create a crawler workflow to extract and catalog metadata from your Cosmos DB for NoSQL accounts, databases, containers, and scripts in Atlan.

Create a Microsoft Azure Cosmos DB for NoSQL crawler workflow to extract and catalog metadata from your accounts, databases, containers, inferred field schemas, stored procedures, triggers, and user-defined functions in Atlan.

## Prerequisites

Before you begin, make sure you have:

* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for connecting data sources to Atlan.
* [Set up Microsoft Azure Cosmos DB for NoSQL](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/set-up-microsoft-azure-cosmos-db-for-nosql/llms.txt) and registered a Microsoft Entra ID service principal with the Cosmos DB Account Reader Role on each account you want to crawl.
* The service principal's **tenant ID**, **client ID**, and **client secret**.

## Create crawler workflow

To create a Microsoft Azure Cosmos DB for NoSQL crawler workflow:

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.
2. Click **Marketplace**.
3. Search for **Cosmos DB for NoSQL** and select it.
4. Click **Install**.
5. Once installation completes, click **Setup Workflow** on the same tile.

If you navigated away before installation completed, go to **New** > **New Workflow** and select **Cosmos DB for NoSQL** to proceed.

### Choose extraction method

For _Extraction method_, choose how Atlan reaches your Cosmos DB accounts. **Direct** is the default.

### Direct

In **Direct** extraction, Atlan Cloud connects to your Cosmos DB accounts over the public Azure endpoint. If your accounts restrict public network access, [allow Atlan's IP range](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/set-up-microsoft-azure-cosmos-db-for-nosql/llms.txt) first.

1. For _Extraction method_, keep **Direct** selected.

2. For _Authentication Type_, the Entra ID service principal option is selected by default. It's the only authentication method this connector supports.

3. Provide the service principal credentials:

 1. For _Tenant ID_, enter the [directory (tenant) ID you copied](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/set-up-microsoft-azure-cosmos-db-for-nosql/llms.txt).

 2. For _Client ID (Application ID)_, enter the [application (client) ID you copied](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/set-up-microsoft-azure-cosmos-db-for-nosql/llms.txt).

 3. For _Client Secret_, enter the [client secret you copied](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/set-up-microsoft-azure-cosmos-db-for-nosql/llms.txt).

4. Click **Test Authentication** to confirm connectivity to Microsoft Azure Cosmos DB.

5. Once authentication is successful, navigate to the bottom of the screen and click **Next**.

### Self-deployed runtime

In **Self-deployed runtime** extraction, the runtime executes metadata extraction within your organization's environment, so your service principal credentials never leave your network.

:::note Self-deployed runtime requires enablement
The **Agent** option appears only when Self-Deployed Runtime is enabled for your tenant. SDR requires additional enablement and licensing—contact your Atlan representative for details.
:::

1. Install Self-Deployed Runtime in your network, if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)

2. For _Extraction method_, select **Agent**, then choose your Self-Deployed Runtime.

3. Store the service principal credentials in the secret store configured with your Self-Deployed Runtime. In agent mode, each credential field becomes a **Store Credential Path** input—enter the path to the secret rather than the secret value itself. For more information, see [Retrieve credentials](https://docs.atlan.com/product/capabilities/build-apps/references/retrieve-credentials).

4. Provide the store credential paths for _Tenant ID_, _Client ID (Application ID)_, and _Client Secret_.

5. Navigate to the bottom of the screen and click **Next**.

### Configure connection

To complete the Microsoft Azure Cosmos DB for NoSQL connection configuration:

1. Provide a _Connection Name_ that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`.
2. To change the users who are able to manage this connection, change the users or groups listed under _Connection Admins_. If you don't specify any user or group, no one can manage the connection, not even admins.
3. Navigate to the bottom of the screen and click **Next** to proceed.

### Configure crawler

Atlan discovers every Cosmos DB for NoSQL account your service principal can access. On the _Metadata_ page, choose which of those accounts and databases to crawl, then run the preflight checks.

- For _Include Filter_, select the accounts and databases to crawl. Leave it empty to crawl everything the service principal can see.
- For _Exclude Filter_, select the accounts and databases to leave out of the crawl. Leave it empty to exclude nothing.

### Direct

In **Direct** extraction, both filters are browsable trees—expand an account to select individual databases within it.

### Self-deployed runtime

In **Self-deployed runtime** extraction there's no in-cluster connection to browse, so both filters are text inputs that accept the same filter as JSON. Each filter maps an account-name regular expression to a list of database-name regular expressions, and an empty list matches all databases in the matched accounts.

For example, the following include filter crawls the `orders` database in the `retail-prod` account:

```json
{"^retail-prod$": ["^orders$"]}
```

Once you've set the filters, run the [preflight checks](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/preflight-checks-for-microsoft-azure-cosmos-db-for-nosql/llms.txt) to validate that Atlan can reach your accounts and read their metadata.

:::note Column extraction is always on
Atlan infers the fields in each container by sampling up to 200 items from it, and publishes nested fields up to 35 levels deep. There's no option to turn this off or change the sample size. Because sampling reads items, it consumes request units on the containers being crawled.
:::

### Run crawler

To run the Microsoft Azure Cosmos DB for NoSQL crawler, after completing the previous steps:

- To run the crawler once, immediately, at the bottom of the screen, click the **Run** button.
- To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the **Schedule & Run** button.

Once the crawler completes running, you can see the assets on Atlan's asset page.

## See also

- [What does Atlan crawl from Microsoft Azure Cosmos DB for NoSQL?](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/what-does-atlan-crawl-from-microsoft-azure-cosmos-db-for-nosql/llms.txt): Learn about the assets and metadata that Atlan discovers and catalogs.
- [Preflight checks for Microsoft Azure Cosmos DB for NoSQL](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/preflight-checks-for-microsoft-azure-cosmos-db-for-nosql/llms.txt): Understand the validations Atlan runs before a crawl.
- [How Atlan connects to Microsoft Azure Cosmos DB for NoSQL](https://docs.atlan.com/llms/connectors/microsoft-azure-cosmos-db-for-nosql/how-atlan-connects-to-microsoft-azure-cosmos-db-for-nosql/llms.txt): Connection protocols, ports, and security.

---