## Automate data profiling

URL: https://docs.atlan.com/product/capabilities/playbooks/how-tos/automate-data-profiling

> Create profiling playbooks to scan data assets at scale, identify data quality issues, and improve asset quality. Setup requires admin user permissions in Atlan.

:::warning
**🤓 Who can do this?** You need to be an [admin user](https://docs.atlan.com/llms/governance/access-control/delegate-administration/llms.txt) in Atlan to create profiling playbooks.
:::

Profiling playbooks power data observability for your assets in Atlan. Scan your assets at scale using profiling playbooks, identify data quality issues or inconsistencies, and improve the overall quality and trust in your data assets.

## Supported sources

Atlan currently supports column profiling for the following connectors:

- [Amazon Athena](https://docs.atlan.com/llms/connectors/amazon-athena/what-does-atlan-crawl-from-amazon-athena/llms.txt)
- [Amazon Redshift](https://docs.atlan.com/llms/connectors/amazon-redshift/what-does-atlan-crawl-from-amazon-redshift/llms.txt)
- [Databricks](https://docs.atlan.com/llms/connectors/databricks/what-does-atlan-crawl-from-databricks/llms.txt)
- [Google BigQuery](https://docs.atlan.com/llms/connectors/google-bigquery/what-does-atlan-crawl-from-google-bigquery/llms.txt)
- [Microsoft SQL Server](https://docs.atlan.com/llms/connectors/microsoft-sql-server/what-does-atlan-crawl-from-microsoft-sql-server/llms.txt)
- [MySQL](https://docs.atlan.com/llms/connectors/mysql/what-does-atlan-crawl-from-mysql/llms.txt)
- [PostgreSQL](https://docs.atlan.com/llms/connectors/postgresql/what-does-atlan-crawl-from-postgresql/llms.txt)
- [Snowflake](https://docs.atlan.com/llms/connectors/snowflake/what-does-atlan-crawl-from-snowflake/llms.txt)
- [Trino](https://docs.atlan.com/llms/connectors/trino/what-does-atlan-crawl-from-trino/llms.txt)

## Create a profiling playbook

To create a profiling playbook:

1. In the left menu in Atlan, click **Governance**.
2. Under the _Governance_ heading of the _Governance center_, click **Playbooks**.
3. To the right of the _Create New_ button, click the downward arrow and then select **Profiling Playbook**.
4. In the _Create new profiling playbook_ dialog, enter the following details:
 1. For _Name_, enter a name for the task to be accomplished - for example, `Tables scan`. (Atlan recommends that the length of a playbook name must be no longer than 46 characters.)
 2. For _Connection_, select a [supported connection](https://docs.atlan.com/llms/governance/playbooks/automate-data-profiling/llms.txt) from the dropdown menu - in this example, we'll select a Google BigQuery connection `development`.
 3. (Optional) For _Description_, enter a description for your playbook.
 4. (Optional) Select an icon for your playbook.
5. Click **Create** to save your playbook.

## Set up rules as filters

:::info **Did you know?** 
 The assets to be scanned are pre-filled based on your selected connection.

:::

To set up rules as filters for your profiling playbook:

1. In the _Build Rules_ page of your profiling playbook, click **Filters**.
2. For the name field, add a name to your filter - for example, _Profiling action_.
3. To set a matching condition for the filters, select **Match all** or **Match any**. _Match all_ will logically `AND` the criteria, while _Match any_ will logically `OR` the criteria.
4. For _Attributes_, select the relevant option. For this example, we'll select _Name_ listed under _Properties_. (Optional) To further refine your asset selection:
 - Click **Connection** to select a specific connection. 
 1. Click **All databases** to filter by databases in a selected connection.
 2. Click **All schemas** to filter by schemas in a selected connection.
 - Click **Connector** to filter assets by [supported connectors](https://docs.atlan.com/llms/catalog/connector-capabilities/supported-sources/llms.txt).
 - Click **Asset type** to filter by specific asset types - for example, tables, columns, queries, glossaries, and more.
 - Click **Certificate** to filter assets by [certification status](https://docs.atlan.com/llms/catalog/discovery/add-certificates/llms.txt).
 - Click **Owners** to filter assets by [asset owners](https://docs.atlan.com/llms/catalog/discovery/add-owners/llms.txt).
 - Click **Tags** to filter assets by your [tags](https://docs.atlan.com/llms/governance/tags/what-are-tags/llms.txt) in Atlan, including imported [Snowflake](https://docs.atlan.com/llms/connectors/snowflake/manage-snowflake-tags/llms.txt) and [dbt](https://docs.atlan.com/llms/connectors/dbt/manage-dbt-tags/llms.txt) tags.
 - (Optional) For [Snowflake tags](https://docs.atlan.com/llms/connectors/snowflake/manage-snowflake-tags/llms.txt) only, to the left of the checkbox, click **Select value**, and then from the _Select tag value_ dialog, select any value(s) to filter assets by tag value.
 - Click **Business Graph, terms, & categories** to filter by a specific [glossary](https://docs.atlan.com/llms/governance/glossary/set-up-glossaries/llms.txt) or [category](https://docs.atlan.com/llms/governance/glossary/what-is-a-glossary/llms.txt) to bulk update all the nested terms or by multiple glossaries and categories.
 - Click **Linked terms** to filter assets by [linked terms](https://docs.atlan.com/llms/governance/glossary/link-terms-to-assets/llms.txt).
 - Click **Schema qualified Name** to filter assets by the qualified name of a given schema.
 - Click **Database qualified Name** to filter assets by the qualified name of a given database.
 - Click **dbt** to filter assets by dbt-specific filters and then select a [dbt Cloud](https://docs.atlan.com/llms/connectors/dbt/what-does-atlan-crawl-from-dbt-cloud/llms.txt) or [dbt Core](https://docs.atlan.com/llms/connectors/dbt/what-does-atlan-crawl-from-dbt-core/llms.txt) filter.
 - Click **Properties** to filter assets by [common asset properties](https://docs.atlan.com/llms/catalog/discovery/use-the-filters-menu/llms.txt).
 - Click **Usage** to filter assets by [usage metrics](https://docs.atlan.com/llms/catalog/usage-and-popularity/find-assets-by-usage/llms.txt).
 - Click **Monte Carlo** to filter assets by [Monte Carlo-specific filters](https://docs.atlan.com/llms/connectors/monte-carlo/what-does-atlan-crawl-from-monte-carlo/llms.txt).
 - Click **Soda** to filter assets by [Soda-specific filters](https://docs.atlan.com/llms/connectors/soda/what-does-atlan-crawl-from-soda/llms.txt).
 - Click **Table/View** to filter tables or views by row count, column count, or size.
 - Click **Column** to filter columns by [column-specific filters](https://docs.atlan.com/llms/catalog/discovery/use-the-filters-menu/llms.txt), including parent asset type or name, data type, or [column keys](https://docs.atlan.com/llms/catalog/connector-capabilities/column-keys-crawled/llms.txt).
 - Click **Process** to filter [lineage processes](https://docs.atlan.com/llms/catalog/lineage/what-are-processes/llms.txt) by the SQL query.
 - Click **Query** to filter assets by associated [visual queries](https://docs.atlan.com/llms/catalog/insights/query-data/llms.txt).
 - Click **Measure** to filter [Microsoft Power BI measures](https://docs.atlan.com/llms/connectors/microsoft-power-bi/what-does-atlan-crawl-from-microsoft-power-bi/llms.txt) using the external measures filter.
5. For _Operator_, select **Is one of** for values to include or **Is not** for values to exclude. Depending on the selected attribute(s), you can also choose from [additional operators](https://docs.atlan.com/llms/catalog/discovery/use-the-filters-menu/llms.txt):
 - Select **Equals (=)** or **Not Equals (!=)** to include or exclude assets through exact match search.
 - Select **Starts With** or **Ends With** to filter assets using the starting or ending sequence of values.
 - Select **Contains** or **Does not contain** to find assets with or without specified values contained within the attribute.
 - Select **Pattern** to filter assets using supported [Elastic DSL regular expressions](https://www.elastic.co/guide/en/elasticsearch/references/current/regexp-syntax.html).
 - Select **Is empty** to filter assets with null values.
6. For _Values_, select the relevant values. The values will vary depending on the selected attributes.
7. (Optional) To add more filters, click **Add filter** and select **Filter** to add individual filters or **Filter** **Group** to nest more filters in a group.
8. (Optional) To view all the assets that match your rules, in the _Filters_ card, click **View** **all** for a preview.

## Confirm profiling actions

:::warning

Column profiling is currently only supported for number and text data types. The profiled column assets will be populated with preconfigured metrics.

:::

To select the actions to be performed based on your rules:

1. The default profiling actions to be performed include:
 - **Base metrics**:
 - Distinct count - number of rows that contain distinct values, relative to the column.
 - Missing count - number of rows that do not contain specific values.
 - **Numeric metrics**:
 - Minimum and maximum values - smallest and greatest values in a numeric column.
 - Average - calculated average of values in a numeric column.
 - Standard deviation - calculated standard deviation of values in a numeric column.
 - Variance - calculated variance of values in a numeric column.
 - **String metrics**:
 - Average length - average length of string values in a column.
 - Minimum and maximum length - minimum and maximum length of string values in a column.
2. Click **Next** to proceed to the next step.
3. In the _Optimize your Profiling query_ popup, the following message will appear: _This Profiling playbook will query `x` rows across `y` assets. To avoid significant computing costs, review your applied filters before proceeding_. Click **Review filters** to review your existing filters or click **Continue anyway** to proceed.

Note that Atlan is working to support sampling functionality in the future.

## Run the playbook

If you'd like to continue working on your playbook, you can save it as a draft. If your playbook is ready, you can proceed to running it.

To run the playbook:

1. You can either:
 - To run the playbook once immediately, click **Run once**.
 - To schedule the playbook to run hourly, daily, weekly, or monthly, click **Schedule** and choose the preferred frequency, timezone, and time.
2. Click **Complete** to confirm your selections.
3. In the resulting screen, click **Go to profile** to view your playbook profile.

Once your playbook run is completed, you will see the data profile updated for your assets! 🎉

## View profiled assets

To view the profiled assets for your playbook:

1. In the _Overview_ page of your playbook, to the right of _Profiling action_, click the total count of profiled assets.
2. In the sidebar to the right, profiled assets will be indicated with a bar graph icon. Click any profiled asset to proceed to viewing profiling data.
3. From the table sidebar, click the **Column** tab to view column assets and then select any of the profiled columns.
4. In the column sidebar to the right, click **Profile** to view profiling data for the selected column asset.

:::info **Did you know?** 
 Once you've created a profiling playbook, you can [monitor, modify, or delete](https://docs.atlan.com/llms/governance/playbooks/manage-playbooks/llms.txt) it at any time.

:::

---