Skip to main content

Ingest data quality metadata

After you configure authentication credentials in Great Expectations Cloud, you can create an ingestion workflow in Atlan to fetch data quality metadata. This workflow maps your Atlan connections to Great Expectations Cloud data sources and enriches your existing assets with data quality insights.

Prerequisites​

Before you begin, make sure you have:

  • Set up Great Expectations Cloud with API authentication credentials
  • At least one connection configured in Atlan that corresponds to a data source in Great Expectations Cloud
  • Log in to your Great Expectations Cloud instance with administrator credentials and fetch the below details:
    • Organization ID: Navigate to your organization settings and note your Organization ID.
    • Workspace ID: Navigate to your workspace settings and note your Workspace ID.
    • API Token: The API token you created in Set up Great Expectations Cloud guide.

Create ingestion workflow​

An ingestion workflow enriches your existing assets with data quality metadata from Great Expectations Cloud. You need to configure authentication credentials and map your Atlan connections to Great Expectations Cloud data sources.

  1. In the top right of any screen in Atlan, click New and then click New workflow.
  2. From the list of packages, select Great Expectations Cloud and then click Setup Workflow.

Configure credentials​

Provide your Great Expectations Cloud credentials to authenticate the workflow.

  1. Enter your Organization ID from the Great Expectations Cloud organization settings.
  2. Enter your Workspace ID from the Great Expectations Cloud workspace settings.
  3. Enter the API token created as part of Set up Great Expectations Cloud.
  4. Click Test Authentication to confirm connectivity to Great Expectations Cloud.
  5. After authentication succeeds, click Next at the bottom of the screen.

Configure connection​

You need to map your Atlan connections to Great Expectations Cloud data sources so that Atlan can enrich the correct assets with data quality metadata.

  1. Select the Atlan connection that contains assets associated with your Great Expectations Cloud data sources from Atlan Connection. This connection must match the data source configured in Great Expectations Cloud.
  2. Select the corresponding data source from the Great Expectations Cloud Connection dropdown list. This list displays the available data sources from your Great Expectations Cloud instance.
  3. To add more mappings, repeat steps 1-2 for additional connections. Each Atlan connection and Great Expectations Cloud data source can only be used once.

Run ingestion​

After configuring your connection mappings, you can run the ingestion workflow to sync data quality metadata from Great Expectations Cloud to your existing assets.

  1. Review your configuration to confirm the mappings are correct.
  2. Choose how to run the ingestion:
    • Click Run to start a one-time sync of data quality metadata
    • Click Schedule & Run to set up automated recurring syncs

After the ingestion completes, your existing Atlan assets are enriched with data quality metadata from Great Expectations Cloud. This includes expectations, validation results, and checkpoint information that appear directly on your table and column assets.

Need help​

If you have questions or need assistance with integration process, reach out to Atlan Support by submitting a request.

See also​