Crawl Microsoft Azure Cosmos DB for NoSQL

||Connect docs via MCP

Create a Microsoft Azure Cosmos DB for NoSQL crawler workflow to extract and catalog metadata from your accounts, databases, containers, inferred field schemas, stored procedures, triggers, and user-defined functions in Atlan.

Prerequisites

Before you begin, make sure you have:

• Reviewed the order of operations for connecting data sources to Atlan.
• Set up Microsoft Azure Cosmos DB for NoSQL and registered a Microsoft Entra ID service principal with the Cosmos DB Account Reader Role on each account you want to crawl.
• The service principal's tenant ID, client ID, and client secret.

Create crawler workflow

To create a Microsoft Azure Cosmos DB for NoSQL crawler workflow:

1. In your Atlan workspace, click Connectors in the left sidebar.
   • If you are using the Old UI (Classic), click New Workflow in the top navigation.
2. Click Marketplace.
3. Search for Cosmos DB for NoSQL and select it.
4. Click Install.
5. Once installation completes, click Setup Workflow on the same tile.

If you navigated away before installation completed, go to New > New Workflow and select Cosmos DB for NoSQL to proceed.

Choose extraction method

For Extraction method, choose how Atlan reaches your Cosmos DB accounts. Direct is the default.

Direct

In Direct extraction, Atlan Cloud connects to your Cosmos DB accounts over the public Azure endpoint. If your accounts restrict public network access, allow Atlan's IP range first.

1. For Extraction method, keep Direct selected.

2. For Authentication Type, the Entra ID service principal option is selected by default. It's the only authentication method this connector supports.

3. Provide the service principal credentials:

   1. For Tenant ID, enter the directory (tenant) ID you copied.

   2. For Client ID (Application ID), enter the application (client) ID you copied.

   3. For Client Secret, enter the client secret you copied.

4. Click Test Authentication to confirm connectivity to Microsoft Azure Cosmos DB.

5. Once authentication is successful, navigate to the bottom of the screen and click Next.

Self-deployed runtime

In Self-deployed runtime extraction, the runtime executes metadata extraction within your organization's environment, so your service principal credentials never leave your network.

Self-deployed runtime requires enablement

The Agent option appears only when Self-Deployed Runtime is enabled for your tenant. SDR requires additional enablement and licensing—contact your Atlan representative for details.

1. Install Self-Deployed Runtime in your network, if you haven't already:

   • Install via Docker Compose
   • Install on Kubernetes

2. For Extraction method, select Agent, then choose your Self-Deployed Runtime.

3. Store the service principal credentials in the secret store configured with your Self-Deployed Runtime. In agent mode, each credential field becomes a Store Credential Path input—enter the path to the secret rather than the secret value itself. For more information, see Retrieve credentials.

4. Provide the store credential paths for Tenant ID, Client ID (Application ID), and Client Secret.

5. Navigate to the bottom of the screen and click Next.

Configure connection

To complete the Microsoft Azure Cosmos DB for NoSQL connection configuration:

1. Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics.
2. To change the users who are able to manage this connection, change the users or groups listed under Connection Admins. If you don't specify any user or group, no one can manage the connection, not even admins.
3. Navigate to the bottom of the screen and click Next to proceed.

Configure crawler

Atlan discovers every Cosmos DB for NoSQL account your service principal can access. On the Metadata page, choose which of those accounts and databases to crawl, then run the preflight checks.

• For Include Filter, select the accounts and databases to crawl. Leave it empty to crawl everything the service principal can see.
• For Exclude Filter, select the accounts and databases to leave out of the crawl. Leave it empty to exclude nothing.

Direct

In Direct extraction, both filters are browsable trees—expand an account to select individual databases within it.

Self-deployed runtime

In Self-deployed runtime extraction there's no in-cluster connection to browse, so both filters are text inputs that accept the same filter as JSON. Each filter maps an account-name regular expression to a list of database-name regular expressions, and an empty list matches all databases in the matched accounts.

For example, the following include filter crawls the orders database in the retail-prod account:

{"^retail-prod$": ["^orders$"]}

Once you've set the filters, run the preflight checks to validate that Atlan can reach your accounts and read their metadata.

Column extraction is always on

Atlan infers the fields in each container by sampling up to 200 items from it, and publishes nested fields up to 35 levels deep. There's no option to turn this off or change the sample size. Because sampling reads items, it consumes request units on the containers being crawled.

Run crawler

To run the Microsoft Azure Cosmos DB for NoSQL crawler, after completing the previous steps:

• To run the crawler once, immediately, at the bottom of the screen, click the Run button.
• To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule & Run button.

Once the crawler completes running, you can see the assets on Atlan's asset page.

See also

• What does Atlan crawl from Microsoft Azure Cosmos DB for NoSQL?: Learn about the assets and metadata that Atlan discovers and catalogs.
• Preflight checks for Microsoft Azure Cosmos DB for NoSQL: Understand the validations Atlan runs before a crawl.
• How Atlan connects to Microsoft Azure Cosmos DB for NoSQL: Connection protocols, ports, and security.