Skip to main content

TransformerInterface

Abstract Class
📁application_sdk.transformers

Abstract base class that defines the contract for all metadata transformers. All transformers must implement the transform_metadata method to convert raw metadata from data sources into Atlas-compatible entities that can be ingested into Atlan. Processes metadata as daft DataFrames and returns transformed daft DataFrames with Atlas-compatible structure.

Methods1

▸

transform_metadata

transform_metadata(self, typename: str, dataframe: daft.DataFrame, workflow_id: str, workflow_run_id: str, entity_class_definitions: Dict[str, Type[Any]] | None = None, **kwargs: Any) -> daft.DataFrame
Transforms raw metadata from data sources into Atlas-compatible entities. Must be implemented by all transformer classes. Converts raw metadata DataFrames into properly structured Atlas entities that can be ingested into Atlan.
Parameters
typenamestr
Required
Type identifier for the metadata (e.g., 'DATABASE', 'SCHEMA', 'TABLE', 'COLUMN')
dataframedaft.DataFrame
Required
Raw metadata as a daft DataFrame
workflow_idstr
Required
Identifier for the workflow requesting the transformation
workflow_run_idstr
Required
Identifier for the specific workflow run
entity_class_definitionsDict[str, Type[Any]] | None
Optional
Optional mapping of entity types to their class definitions
**kwargsAny
Optional
Additional keyword arguments for specific transformer implementations
Returns
daft.DataFrame - Transformed metadata as a daft DataFrame with Atlas-compatible structure

Transformer implementations​

The Application SDK provides two concrete implementations of the TransformerInterface, each optimized for different transformation approaches. All implementations follow the transformer interface contract, which defines the transform_metadata method that converts raw metadata into Atlas-compatible entities.

🗺️

AtlasTransformer

PyAtlan Classes

Converts metadata into Atlas entities using pyatlan library classes. Uses entity class definitions to create properly structured Atlas entities with relationships and qualified names.

📝

QueryBasedTransformer

YAML Templates

Transforms metadata using SQL queries defined in YAML templates. Executes SQL transformations on raw dataframes using the daft engine and automatically creates nested structures.

Common utilities​

The transformers module provides utility functions for common operations including text processing, qualified name building, and template path mapping.

See Common utilities for complete reference documentation.

Usage in workflows​

Transformers are typically used in metadata extraction workflows:

from application_sdk.transformers import AtlasTransformer
from application_sdk.activities.metadata_extraction import BaseSQLMetadataExtractionActivities

class MyActivities(BaseSQLMetadataExtractionActivities):
    transformer_class = AtlasTransformer

    def __init__(self):
        super().__init__(transformer_class=AtlasTransformer)

The transformer is automatically initialized with:

  • connector_name: Application name
  • tenant_id: Tenant identifier
  • Additional parameters from workflow configuration

Entity types​

Common entity types supported by transformers:

  • DATABASE: Database entities
  • SCHEMA: Schema entities
  • TABLE: Table entities
  • VIEW: View entities
  • MATERIALIZED VIEW: Materialized view entities
  • COLUMN: Column entities
  • FUNCTION: Function entities
  • PROCEDURE: Stored procedure entities
  • TAG_REF: Tag attachment entities

See also​