Manage data products
Create new data product
To create a new data product:
- Java
- Python
- Kotlin
- Raw REST API
FluentSearch assets = Table.select(client) // (1)
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED))
.where(Table.ATLAN_TAGS.eq("Marketing"))
.build();
DataProduct dp = DataProduct.creator("Marketing Influence", // (2)
DataDomain.refByQualifiedName("default/domain/marketing"), // (3)
assets) // (4)
.build(); // (5)
AssetMutationResponse response = dp.save(client); // (6)
- When defining a data product, you must define the assets within it. These are defined through a search, so that the assets included can be automatically managed. In this example, we're selecting all verified tables that have a tag of
Marketing. - You must provide a human-readable name for your data product.
- You must also provide the domain in which the data product should exist.
- And finally the search that was defined earlier, to define which assets to include in the data product.
- You then need to build the object.
- And then you can
save()the object you've built to create the new data product in Atlan. Because this operation will persist the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import DataProduct, Table
from pyatlan.model.fluent_search import CompoundQuery, FluentSearch
from pyatlan.model.enums import CertificateStatus
client = AtlanClient()
assets = (
FluentSearch()
.where(CompoundQuery.active_assets())
.where(CompoundQuery.asset_type(Table))
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED.value))
.where(Table.ATLAN_TAGS.eq("Marketing"))
).to_request() # (1)
product = DataProduct.creator(
name="Marketing Influence", # (2)
asset_selection=assets, # (3)
domain_qualified_name="default/domain/marketing" # (4)
)
response = client.asset.save(product) # (5)
- When defining a data product, you must define the assets within it. These are defined through a search, so that the assets included can be automatically managed. In this example, we're selecting all verified tables that have a tag of
Marketing. - You must provide a human-readable name for your data product.
- You must provide the index search request that was defined earlier, to define which assets to include in the data product.
- You must also provide the domain in which the data product should exist.
- And then you can
save()the object you've built to create the new data product in Atlan.
val assets = Table.select(client) // (1)
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED))
.where(Table.ATLAN_TAGS.eq("Marketing"))
.build()
val dp = DataProduct.creator("Marketing Influence", // (2)
DataDomain.refByQualifiedName("default/domain/marketing"), // (3)
assets) // (4)
.build() // (5)
val response = dp.save(client) // (6)
- When defining a data product, you must define the assets within it. These are defined through a search, so that the assets included can be automatically managed. In this example, we're selecting all verified tables that have a tag of
Marketing. - You must provide a human-readable name for your data product.
- You must also provide the domain in which the data product should exist.
- And finally the search that was defined earlier, to define which assets to include in the data product.
- You then need to build the object.
- And then you can
save()the object you've built to create the new data product in Atlan. Because this operation will persist the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
{
"entities": [
{
"typeName": "DataProduct", // (1)
"attributes": {
"name": "Marketing Influence", // (2)
"qualifiedName": "default/domain/8S8vbC9hT6iPLMY2ydXqf/super/product/marketingInfluence", // (3)
"parentDomainQualifiedName": "default/domain/8S8vbC9hT6iPLMY2ydXqf/super", // (4)
"superDomainQualifiedName": "default/domain/8S8vbC9hT6iPLMY2ydXqf/super", // (5)
"dataProductAssetsDSL": "{\"query\": {\"attributes\": [\"__traitNames\", \"connectorName\", \"__customAttributes\", \"certificateStatus\", \"tenantId\", \"anchor\", \"parentQualifiedName\", \"Query.parentQualifiedName\", \"AtlasGlossaryTerm.anchor\", \"databaseName\", \"schemaName\", \"parent\", \"connectionQualifiedName\", \"collectionQualifiedName\", \"announcementMessage\", \"announcementTitle\", \"announcementType\", \"announcementUpdatedAt\", \"announcementUpdatedBy\", \"allowQuery\", \"allowQueryPreview\", \"adminGroups\", \"adminRoles\", \"adminUsers\", \"category\", \"credentialStrategy\", \"connectionSSOCredentialGuid\", \"certificateStatus\", \"certificateUpdatedAt\", \"certificateUpdatedBy\", \"classifications\", \"connectionId\", \"connectionQualifiedName\", \"connectorName\", \"dataType\", \"defaultDatabaseQualifiedName\", \"defaultSchemaQualifiedName\", \"description\", \"displayName\", \"links\", \"link\", \"meanings\", \"name\", \"ownerGroups\", \"ownerUsers\", \"qualifiedName\", \"typeName\", \"userDescription\", \"displayDescription\", \"subDataType\", \"rowLimit\", \"queryTimeout\", \"previewCredentialStrategy\", \"policyStrategy\", \"policyStrategyForSamplePreview\", \"useObjectStorage\", \"objectStorageUploadThreshold\", \"outputPortDataProducts\"], \"dsl\": {\"from\": 0, \"query\": {\"bool\": {\"filter\": {\"bool\": {\"filter\": [{\"term\": {\"__state\": {\"value\": \"ACTIVE\", \"case_insensitive\": false}}}, {\"term\": {\"__typeName.keyword\": {\"value\": \"Table\", \"case_insensitive\": false}}}, {\"term\": {\"certificateStatus\": {\"value\": \"VERIFIED\", \"case_insensitive\": false}}}, {\"term\": {\"__traitNames\": {\"value\": \"Marketing\", \"case_insensitive\": false}}}]}}}}}, \"suppressLogs\": true}, \"filterScrubbed\": true}",
// (6)
"dataProductAssetsPlaybookFilter": "{\"condition\":\"AND\",\"isGroupLocked\":false,\"rules\":[]}"
// (7)
},
"relationshipAttributes": {
"dataDomain": {
"typeName": "DataDomain",
"uniqueAttributes": { // (8)
"qualifiedName": "default/domain/8S8vbC9hT6iPLMY2ydXqf/super"
}
}
}
}
]
}
-
The
typeNamemust be exactlyDataProduct. -
Provide a human-readable name for your data product.
-
Make sure the
qualifiedNamefollows the pattern:<parentDomainQualifiedName>/product/<lowerCamelCaseName>. -
Provide a
parentDomainQualifiedNamefor the data domain under which you want to create this product. -
Provide a
superDomainQualifiedNamefor the data domain under which you want to create this product. If creating a product under sub-domains, this should be the qualified name of the root-level domain. -
Provide the DSL that defines the assets to include in the data product as an embedded JSON string.
Use SDK to create data products
The above data products assets DSL requires a filter as a nested object construct within an outer bool, rather than a list or array. It's recommended to create data products via SDK as it handles this complexity automatically.
:::
7. Specify the default playbook filter to define which assets are shown in the data product UI.
8. Specify the qualifiedName of the data domain under which you want to create this product.
Retrieve data product
To retrieve a data product by its human-readable name:
- Java
- Python
- Kotlin
- Raw REST API
DataProduct product = DataProduct.findByName( // (1)
client, "marketingInfluence", List.of("certificateStatus")
).get(0);
-
Use
DataProduct.findByName()method to retrieve a data product by its human-readable name:- client through which to access a tenant.
- name of the data product.
- (optional) a list of attributes to retrieve for the data product, for example
certificateStatus.
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import DataProduct
client = AtlanClient()
product = client.asset.find_product_by_name( # (1)
name="marketingInfluence",
attributes=["certificateStatus"]
)
assert product
assert product.certificate_status
-
Use
client.asset.find_product_by_name()method to retrieve a data product by its human-readable name:- name of the data product.
- (optional) a list of attributes to retrieve
for the data product, for example
certificateStatus.
val product = DataProduct.findByName( // (1)
client, "marketingInfluence", listOf("certificateStatus")
).get(0)
-
Use
DataProduct.findByName()method to retrieve a data product by its human-readable name:- client through which to access a tenant.
- name of the data product.
- (optional) a list of attributes to retrieve for the data product, for example
certificateStatus.
{
"dsl": {
"from": 0,
"size": 100,
"aggregations": {},
"track_total_hits": true,
"query": {
"bool": {
"filter": [
{
"term": {
"name.keyword": {
"value": "marketingInfluence" // (1)
}
}
},
{
"term": {
"__typeName.keyword": {
"value": "DataProduct"
}
}
}
]
}
},
"sort": [
{
"__guid": {
"order": "asc"
}
}
]
},
"attributes": [
"certificateStatus" // (2)
]
}
- Human-readable name of the data product.
- (optional) a list of attributes to retrieve
for the data product, for example
certificateStatus.
Retrieve all assets linked to data product
To retrieve list of all assets associated with a data product:
- Java
- Python
- Kotlin
- Raw REST API
DataProduct dp = DataProduct.get(client,
"49abc625-9a03-4733-bdfb-ec0cb5315cac", true); // (1)
IndexSearchResponse response = dp.getAssets(client); // (2)
for (Asset result : response)
- Use
DataProduct.get()method to retrieve the data product by itsGUID. - Use
DataProduct.getAssets()method to retrieve all assets linked to the data product, providing theclientused to access the tenant.
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import DataProduct, Asset
client = AtlanClient()
product = client.asset.get_by_guid( # (1)
guid="49abc625-9a03-4733-bdfb-ec0cb5315cac",
asset_type=DataProduct,
attributes=[DataProduct.DATA_PRODUCT_ASSETS_DSL],
related_attributes=[Asset.DESCRIPTION, Asset.USER_DESCRIPTION]
)
response = product.get_assets(client) # (2)
for assets in response:
// Do something with the assets retrieved...
assert product
assert response
- Use
client.asset.get_by_guid()method to retrieve the data product by itsGUID. - Use
DataProduct.get_assets()method to retrieve all assets linked to the data product.
val product = DataProduct.get(client,
"49abc625-9a03-4733-bdfb-ec0cb5315cac", true); // (1)
val response = dp.getAssets(client); // (2)
for (result in response)
- Use
DataProduct.get()method to retrieve the data product by itsGUID. - Use
DataProduct.getAssets()method to retrieve all assets linked to the data product, providing theclientused to access the tenant.
{
"dsl": {
"from": 0,
"size": 100,
"aggregations": {},
"track_total_hits": true,
"query": {
"bool": {
"filter": {
"bool": {
"filter": [ // (1)
}
},
{
"term": {
"__typeName.keyword": {
"value": "Table",
"case_insensitive": false
}
}
}
]
}
}
}
},
"sort": []
},
"attributes": [ // (2)
"certificateStatus","connectorName",
"__customAttributes", "certificateStatus",
]
}
- List the asset type and active status for all assets included in the data product.
- (optional) a list of attributes to retrieve
for the data product's list of all assets, for example
certificateStatus.
Update data product
To update a data product:
- Java
- Python
- Kotlin
- Raw REST API
FluentSearch assets = Table.select(client)
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED))
.where(Table.ATLAN_TAGS.eq("Digital Marketing"))
.build(); // (1)
DataProduct dp = DataProduct.updater("default/product/marketingInfluence", // (2)
"Marketing Influence")
.userDescription("Now with a description!") // (3)
.assetSelection(Atlan.getDefaultClient(), assets)
.build(); // (4)
AssetMutationResponse response = dp.save(client); // (5)
- (Optional) You can also update the assets within an existing product.
These assets are defined through a search, allowing for automatic management.
In this example, we select all verified tables that are tagged with
Digital Marketing. - Use the
updater()method to update a data product, providing thequalifiedNameandnameof the data product. - You can chain additional enrichments onto the updater, such as:
userDescription: updating the product's description.assetSelection: modifying the assets within the product.
- You then need to build the object.
- You can then
save()the object you've built to update the data product in Atlan. Because this operation will persist the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import DataProduct
client = AtlanClient()
assets = (
FluentSearch()
.where(CompoundQuery.active_assets())
.where(CompoundQuery.asset_type(Table))
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED.value))
.where(Table.ATLAN_TAGS.eq("Digital Marketing"))
).to_request() # (1)
product = DataProduct.updater( # (2)
qualified_name="default/product/marketingInfluence", # (3)
name="Marketing Influence", # (4)
asset_selection=assets, # (5)
)
product.user_description = "Now with a description!" # (6)
response = client.asset.save(product) # (7)
- (Optional) You can also update the assets within an existing product.
These assets are defined through a search, allowing for automatic management.
In this example, we select all verified tables that are tagged with
Digital Marketing. - Use the
updater()method to update a data product. - You must provide the
qualifiedNameof the data product. - You must provide the
nameof the data product. - Optionally, you can provide an index search request to define the assets to include in the data product.
- You can then add on any other updates, such as changing the user description of the data product.
- To update the data product in Atlan, call the
save()method with the object you've built.
val assets = Table.select(client)
.where(Table.CERTIFICATE_STATUS.eq(CertificateStatus.VERIFIED))
.where(Table.ATLAN_TAGS.eq("Digital Marketing"))
.build(); // (1)
val dp = DataProduct.updater("default/product/marketingInfluence", // (2)
"Marketing Influence")
.userDescription("Now with a description!") // (3)
.assetSelection(Atlan.getDefaultClient(), assets)
.build() // (4)
val response = dp.save(client) // (5)
- (Optional) You can also update the assets within an existing product.
These assets are defined through a search, allowing for automatic management.
In this example, we select all verified tables that are tagged with
Digital Marketing. - Use the
updater()method to update a data product, providing thequalifiedNameandnameof the data product. - You can chain additional enrichments onto the updater, such as:
userDescription: updating the product's description.assetSelection: modifying the assets within the product.
- You then need to build the object.
- You can then
save()the object you've built to update the data product in Atlan. Because this operation will persist the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
{
"entities": [
{
"typeName": "DataProduct", // (1)
"attributes": {
"name": "Marketing Influence", // (2)
"qualifiedName": "default/product/marketingInfluence", // (3)
"userDescription": "Now with a description!", // (4)
"dataProductAssetsDSL": "{\"query\": {\"attributes\": [\"__traitNames\", \"connectorName\", \"__customAttributes\", \"certificateStatus\", \"tenantId\", \"anchor\", \"parentQualifiedName\", \"Query.parentQualifiedName\", \"AtlasGlossaryTerm.anchor\", \"databaseName\", \"schemaName\", \"parent\", \"connectionQualifiedName\", \"collectionQualifiedName\", \"announcementMessage\", \"announcementTitle\", \"announcementType\", \"announcementUpdatedAt\", \"announcementUpdatedBy\", \"allowQuery\", \"allowQueryPreview\", \"adminGroups\", \"adminRoles\", \"adminUsers\", \"category\", \"credentialStrategy\", \"connectionSSOCredentialGuid\", \"certificateStatus\", \"certificateUpdatedAt\", \"certificateUpdatedBy\", \"classifications\", \"connectionId\", \"connectionQualifiedName\", \"connectorName\", \"dataType\", \"defaultDatabaseQualifiedName\", \"defaultSchemaQualifiedName\", \"description\", \"displayName\", \"links\", \"link\", \"meanings\", \"name\", \"ownerGroups\", \"ownerUsers\", \"qualifiedName\", \"typeName\", \"userDescription\", \"displayDescription\", \"subDataType\", \"rowLimit\", \"queryTimeout\", \"previewCredentialStrategy\", \"policyStrategy\", \"policyStrategyForSamplePreview\", \"useObjectStorage\", \"objectStorageUploadThreshold\", \"outputPortDataProducts\"], \"dsl\": {\"from\": 0, \"query\": {\"bool\": {\"filter\": {\"bool\": {\"filter\": [{\"term\": {\"__state\": {\"value\": \"ACTIVE\", \"case_insensitive\": false}}}, {\"term\": {\"__typeName.keyword\": {\"value\": \"Table\", \"case_insensitive\": false}}}, {\"term\": {\"certificateStatus\": {\"value\": \"VERIFIED\", \"case_insensitive\": false}}}, {\"term\": {\"__traitNames\": {\"value\": \"Digital Marketing\", \"case_insensitive\": false}}}]}}}}}, \"suppressLogs\": true}, \"filterScrubbed\": true}",
// (5)
}
}
]
}
-
The
typeNamemust be exactlyDataProduct. -
Human-readable name for your data product.
-
You must provide the the
qualifiedNameof the data product to update. -
You can add on any other updates, such as changing the user description of the data product.
-
(Optional) You can update the product's assets by providing a DSL that defines the assets to include in the data product, as an embedded JSON string.
Use SDK to update data products
The above data products assets DSL requires a filter as a nested object construct within an outer bool, rather than a list or array. It's recommended to update data products via SDK as it handles this complexity automatically. :::
Delete data product
Soft-delete (archive)
To soft-delete, or archive, a data product:
- Java
- Python
- Kotlin
- Raw REST API
AssetDeletionResponse response = DataProduct.delete(client, "218c8144-dc39-43a5-b0c0-9eeb4d11e74a"); // (1)
- To archive a data product in Atlan, call the
DataProduct.delete()method with the GUID of the data product. Because this operation will archive the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
from pyatlan.client.atlan import AtlanClient
client = AtlanClient()
client.asset.delete_by_guid("218c8144-dc39-43a5-b0c0-9eeb4d11e74a") # (1)
- To archive a data product in Atlan, call the
asset.delete_by_guid()method with the GUID of the data product.
val response = DataProduct.delete(client, "218c8144-dc39-43a5-b0c0-9eeb4d11e74a") // (1)
- To archive a data product in Atlan, call the
DataProduct.delete()method with the GUID of the data product. Because this operation will archive the asset in Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
// (1)
- All the details for deleting the data product are specified in the URL directly. Note that you must provide the GUID of the data product to delete it.
Hard-delete (purge)
To permanently delete (purge) a data product:
- Java
- Python
- Kotlin
- Raw REST API
AssetDeletionResponse response = DataProduct.purge(client, "218c8144-dc39-43a5-b0c0-9eeb4d11e74a"); // (1)
- To permanently delete a data product in Atlan, call the
DataProduct.purge()method with the GUID of the data product. Because this operation will remove the asset from Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
from pyatlan.client.atlan import AtlanClient
client = AtlanClient()
client.asset.purge_by_guid("218c8144-dc39-43a5-b0c0-9eeb4d11e74a") # (1)
- To permanently delete a data product in Atlan, call the
asset.purge_by_guid()method with the GUID of the data product.
val response = DataProduct.purge(client, "218c8144-dc39-43a5-b0c0-9eeb4d11e74a") // (1)
- To permanently delete a data product in Atlan, call the
DataProduct.purge()method with the GUID of the data product. Because this operation will remove the asset from Atlan, you must provide it anAtlanClientthrough which to connect to the tenant.
// (1)
- All the details for deleting the data product are specified in the URL directly. Note that you must provide the GUID of the data product to delete it.