## Crawl Confluent Schema Registry

URL: https://docs.atlan.com/apps/connectors/schema/confluent-schema-registry/how-tos/crawl-confluent-schema-registry

> Configure and run the standalone Confluent Schema Registry crawler in Atlan.

Configure and run the standalone Schema Registry crawler to extract subjects and schema versions into Atlan. To crawl Schema Registry alongside Confluent Kafka in a single workflow, see [Crawl Confluent Kafka](https://docs.atlan.com/llms/connectors/confluent-kafka/crawl-confluent-kafka/llms.txt) instead.

## Prerequisites

Before you begin, make sure you have:

* Completed the [Confluent Schema Registry setup](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt) and have your Schema Registry endpoint, API key, and API secret ready
* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt) for crawling metadata
* The required permissions in Atlan to create and manage a connection

## Create crawler workflow

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Confluent Schema Registry Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure extraction

Choose your extraction method:

### Direct

1. For _Host_, enter your [schema registry endpoint](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt).

2. For _API Key_, enter the [API key you copied](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt).

3. For _API Secret_, enter the [API secret you copied](https://docs.atlan.com/llms/connectors/confluent-schema-registry/set-up-confluent-schema-registry/llms.txt).

4. Click **Test Authentication** to confirm connectivity, then click **Next**.

### Agent

Before configuring the crawler:

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)
2. Confirm the runtime can reach your Confluent Schema Registry over your local network and that [network security](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-network-security/llms.txt) is configured.

To configure the crawler:

1. Select the **Agent** tab in the workflow setup.

2. Under _Secure Agent Configuration_, select your deployed agent from the **Agent** dropdown and the secret store from the **Secret Store** dropdown.

3. For _Host_, enter your schema registry endpoint as reachable from within your network.

4. For _API Key_, reference the secret store path where the API key is stored.

5. For _API Secret_, reference the secret store path where the API secret is stored.

6. Store sensitive credential values in your secret store and reference them in the corresponding fields. For more information, see [Configure secrets for workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

7. Click **Next** after completing the configuration.

### Configure connection

1. Provide a _Connection Name_ that represents your source environment -- for example, `production`, `development`, `gold`, or `analytics`.

2. Under _Connection Admins_, add the users or groups that can manage this connection.

 :::warning

 If you don't specify any user or group, no one can manage the connection -- not even admins.

 :::

3. At the bottom of the screen, click **Next**.

### Configure crawling options

On the _Metadata_ page, you can override the defaults for any of these options:

- Click **Exclude subjects** to exclude specific subjects from crawling. Defaults to no exclusions if none are specified.
- Click **Include subjects** to limit crawling to specific subjects. Defaults to all subjects if none are specified.

If an asset appears in both include and exclude filters, the exclude filter takes precedence.

### Run crawler

### Direct

1. Click **Preflight checks** to validate [permissions and configuration](https://docs.atlan.com/llms/connectors/confluent-schema-registry/preflight-checks-for-confluent-schema-registry/llms.txt) before running.

2. After preflight checks pass, either:
 - Click **Run** to run the crawler once immediately.
 - Click **Schedule & Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

### Agent

Either:

- Click **Run** to run the crawler once immediately.
- Click **Schedule & Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

Once the crawler completes, the assets appear on Atlan's asset page.

## See also

- [What does Atlan crawl from Confluent Schema Registry](https://docs.atlan.com/llms/connectors/confluent-schema-registry/what-does-atlan-crawl-from-confluent-schema-registry/llms.txt): Assets and metadata discovered during crawling
- [Preflight checks for Confluent Schema Registry](https://docs.atlan.com/llms/connectors/confluent-schema-registry/preflight-checks-for-confluent-schema-registry/llms.txt): Validation checks for permissions and configuration

---