## Crawl PostgreSQL

URL: https://docs.atlan.com/apps/connectors/database/postgresql/how-tos/crawl-postgresql

> Configure and run PostgreSQL metadata extraction workflows to catalog databases, schemas, tables, views, and columns in Atlan.

Extract metadata assets from your PostgreSQL database into Atlan.

## Prerequisites

Before you begin, verify you have:

* [Configured the PostgreSQL permissions](https://docs.atlan.com/llms/connectors/postgresql/set-up-postgresql/llms.txt) (and [set up a private network link](https://docs.atlan.com/llms/connectors/postgresql/set-up-a-private-network-link-to-postgresql/llms.txt) if using private network)
* Access to your PostgreSQL instance and credentials
* Reviewed the [order of operations](https://docs.atlan.com/llms/catalog/connector-capabilities/order-workflows/llms.txt)

## Create crawler workflow

Create a new workflow and select PostgreSQL as your connector source.

1. In your Atlan workspace, click **Connectors** in the left sidebar.
 * If you are using the **Old UI (Classic)**, click **New Workflow** in the top navigation.

2. Click **Marketplace**.

3. Search for **Postgres Assets** and select it.

4. Click **Install**.

5. Once installation completes, click **Setup Workflow** on the same tile.

### Configure extraction

When setting up metadata extraction from your PostgreSQL database, choose how Atlan connects and extracts metadata. Select the extraction method that best fits your organization's security and network requirements:

:::warning Offline extraction sunset
The offline extraction method has been sunset and is no longer available. For on-premises or network-restricted environments, use the **Agent** extraction method with [Self-Deployed Runtime](https://docs.atlan.com/llms/platform/self-deployed-runtime/llms.txt).
:::

### Direct

Atlan SaaS connects directly to your PostgreSQL database. This method supports Basic and IAM authentication and lets you test the connection before proceeding.

1. For **Host**, enter the host for your PostgreSQL instance.

2. For **Port**, enter the port number of your PostgreSQL instance (default: 5432).

3. Choose an authentication type that matches your PostgreSQL configuration:

1. Select **Basic** authentication.

2. For **Username**, enter the username you configured in PostgreSQL.

3. For **Password**, enter the password for the username.

### IAM User authentication

1. Select **IAM User** authentication.

2. For **AWS Access Key**, enter the AWS access key.

3. For **AWS Secret Key**, enter the AWS secret key.

4. For **Username**, enter the database username you configured.

### IAM Role authentication

1. Select **IAM Role** authentication.

2. For **AWS Role ARN**, enter the ARN of the [role you created](https://docs.atlan.com/llms/connectors/postgresql/set-up-postgresql/llms.txt).

3. For **Username**, enter the database username you configured.

4. For **AWS External ID**, enter the external ID if you configured one in the [role definition](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).

4. For **Database**, enter the name of the database to crawl.

5. Click **Test Authentication** to confirm connectivity to PostgreSQL. When successful, click **Next** to proceed with the connection configuration.

### Agent

Self-Deployed Runtime runs within your organization and connects to your PostgreSQL database. This method keeps connections inside your network perimeter.

1. Install Self-Deployed Runtime if you haven't already:
 - [Install via Docker Compose](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-docker-compose/llms.txt)
 - [Install on Kubernetes](https://docs.atlan.com/llms/platform/self-deployed-runtime/install-on-kubernetes/llms.txt)

2. Select the **Agent** tab and configure the PostgreSQL data source by adding the secret keys for your secret store.

3. Choose an authentication type that matches your PostgreSQL configuration:

1. Select **Basic** authentication.

2. For **Host**, enter the host for your PostgreSQL instance.

3. For **Port**, enter the port number of your PostgreSQL instance (default: 5432).

4. For **Username**, enter the username you configured in PostgreSQL (or reference the secret key where it's stored).

5. For **Password**, enter the password for the username (or reference the secret key where it's stored).

### IAM User authentication

1. Select **IAM User** authentication.

2. For **Host**, enter the host for your PostgreSQL instance.

3. For **Port**, enter the port number of your PostgreSQL instance (default: 5432).

4. For **AWS Access Key**, enter the AWS access key (or reference the secret key where it's stored).

5. For **AWS Secret Key**, enter the AWS secret key (or reference the secret key where it's stored).

6. For **Username**, enter the database username you configured.

### IAM Role authentication

1. Select **IAM Role** authentication.

2. For **Host**, enter the host for your PostgreSQL instance.

3. For **Port**, enter the port number of your PostgreSQL instance (default: 5432).

4. For **AWS Role ARN**, enter the ARN of the [role you created](https://docs.atlan.com/llms/connectors/postgresql/set-up-postgresql/llms.txt) (or reference the secret key where it's stored).

5. For **Username**, enter the database username you configured.

6. For **AWS External ID**, enter the external ID if you configured one in the [role definition](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).

4. For **Database**, enter the name of the database to crawl.

5. Store sensitive information in your secret store and reference the secrets in the corresponding fields. For more information, see [Configure secrets for workflow execution](https://docs.atlan.com/llms/platform/self-deployed-runtime/configure-workflow-execution/llms.txt).

6. Click **Next** after completing the configuration.

### Configure connection

Set up the connection name and access controls for your PostgreSQL data source in Atlan.

1. Provide a **Connection Name** that represents your source environment. For example, you might use values like `production`, `development`, `gold`, or `analytics`.

2. To change the users able to manage this connection, update the users or groups listed under **Connection Admins**. If you don't specify any user or group, nobody can manage the connection (not even admins).

3. At the bottom of the screen, click **Next** to proceed.

### Configure crawler

Before running the crawler, you can configure which assets to include or exclude:

- To exclude specific assets from crawling, click **Exclude Metadata**. This defaults to no assets if none are specified.
- To include specific assets in crawling, click **Include Metadata**. This defaults to all assets if none are specified.
- To have the crawler ignore tables and views based on a naming convention, specify a regular expression in the **Exclude regex for tables & views** field.
- For **Advanced Config**, keep **Default** for the default configuration or click **Custom** to configure the crawler:
 - For **Enable Source Level Filtering**, click **True** to enable schema-level filtering at source or click **False** to disable it.
 - For **Use JDBC Internal Methods**, click **True** to enable JDBC internal methods for data extraction or click **False** to disable it.

If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

### Run crawler

### Direct

1. Click **Preflight checks** to validate permissions and configuration before running the crawler. This helps identify any [potential issues](https://docs.atlan.com/llms/connectors/postgresql/preflight-checks-for-postgresql/llms.txt) early.

2. After the preflight checks pass, you can either:
 - Click **Run** to run the crawler once immediately.
 - Click **Schedule Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

### Agent

You can either:

- Click **Run** to run the crawler once immediately.
- Click **Schedule Run** to schedule the crawler to run hourly, daily, weekly, or monthly.

Once the crawler has completed running, you can see the assets on Atlan's asset page.

## See also

- [How Atlan connects to PostgreSQL](https://docs.atlan.com/llms/connectors/postgresql/how-atlan-connects-to-postgresql/llms.txt): Connectivity, authentication, and data access patterns
- [What does Atlan crawl from PostgreSQL](https://docs.atlan.com/llms/connectors/postgresql/what-does-atlan-crawl-from-postgresql/llms.txt): Metadata and assets discovered during crawling
- [Preflight checks for PostgreSQL](https://docs.atlan.com/llms/connectors/postgresql/preflight-checks-for-postgresql/llms.txt): Verify prerequisites before crawling
- [Troubleshooting PostgreSQL connectivity](https://docs.atlan.com/llms/connectors/postgresql/troubleshooting-postgresql-connectivity/llms.txt): Resolve common connection issues

---