## Data Model Ingestion

URL: https://docs.atlan.com/product/capabilities/data-models/references/data-model-ingestion

> Reference guide for ingesting data models into Atlan

<!--vale off-->

# Data Model Ingestion 

<!--vale on-->

The **Data Model Ingestion** app loads logical and relational metadata from an input file into Atlan. It’s designed to help teams define how their data is structured, organized, and related within Atlan without manual setup. 

Files can be provided either by uploading them directly from your local machine or by fetching them from a cloud object store. Supported object storage systems include Amazon S3, Google Cloud Storage (GCS), and Azure Data Lake Storage (ADLS). 

This reference provides complete configuration details for data model ingestion, including input handling rules and the structure of the required input file. 

## Access 

The **Data Model Ingestion** app isn't enabled by default. To use this app, contact [Atlan support](https://docs.atlan.com/support/submit-request) and request it be added to your tenant. Once enabled, data model ingestion workflows can be set up and run by admins or users with workflow permissions. 

## Source 

This section defines how the input file for data model ingestion is provided and identified in Atlan. 

#### Workflow name 

Specifies the display name for the workflow in Atlan. This name is used to identify the ingestion job in the UI and logs. Choose a name that clearly reflects the purpose or scope of the import. 

<dd>

*Example:* 

```text
Production Finance Data Model
```

</dd>

## Import metadata from 

This property defines how the file containing the data model is provided to the workflow. The file must match the [Data Model file format](#excel-input). 

There are two ways to provide the file: 

- **Direct file uploads:** Upload a Excel file directly from your local machine. This is useful for smaller files or ad-hoc imports. See [Direct file uploads](#direct-file-uploads). 
- **Object storage:** Fetch the file from a supported cloud object store (S3, GCS, or ADLS). This is recommended for larger files or recurring imports. See [Object storage](#object-storage). 

### Direct file uploads 

Upload a file directly from your local machine. This option is best for smaller files or ad-hoc imports that are run manually. 

:::warning File upload limits 
Direct file uploads are limited to a maximum file size of **20 MB**, with a rate limit of **10 uploads per minute per client**, and support only one file per run. Executable file types such as `.exe`, `.bat`, `.sh`, `.ps1`, `.cmd`, `.com`, `.dll`, and `.so` are blocked. For larger or recurring imports, use [object storage](#object-storage). 
::: 

### Object storage 

This property imports the data model file from a cloud object store rather than a local upload. It's recommended for large files and recurring ingestion jobs. Supported providers are Amazon S3, Google Cloud Storage (GCS), and Azure Data Lake Storage (ADLS). When this option is selected, additional storage-specific properties such as bucket, project ID, or container become available. 

### Amazon S3

Amazon S3 enables you to store and retrieve objects at scale. You can use this option when the data model Excel file is stored in an S3 bucket. 

#### AWS access key

The access key for your AWS account. You can find this in the **AWS Management Console > IAM > Users > Security credentials** tab.

- **Required** if you are using access/secret key authentication.
- **Keep empty** if you are using tenant-backed, cross-account, or role-based authentication.

*Example:*

```
AKIAIOSFODNN7EXAMPLE
```

#### AWS secret key

The secret key that pairs with your access key. This is generated when you create an access key in IAM. You must download it at creation time or rotate and generate a new one if lost.

- **Required** if you are using access/secret key authentication.
- **Keep empty** if you are using tenant-backed, cross-account, or role-based authentication.

*Example:*

```
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```

#### Region

The AWS region in which your bucket is located (for example, `us-east-1`). You can find this in the **S3 service dashboard** when selecting your bucket.

- **Required** if you are using access/secret key authentication.
- **Keep empty** if you are using tenant-backed, cross-account, or role-based authentication (region is inferred automatically).

*Example:*

```
ap-southeast-1
```

#### Bucket

The name of the S3 bucket that contains your data model file. The bucket name is listed in the **S3 service dashboard**.

- **Required** in all scenarios (cross-account, access/secret keys, and role-based).
- **Keep empty** if you are using tenant-backed authentication and don't know the specific bucket name.

*Example:*

```
my-company-data-models
```

#### Prefix 

Optional. Directory path within the bucket where your data model file is stored. 

*Example:*

```
data-models/sales/
```

#### Object key 

The file name (including extension) of your data model file within the bucket or prefix. 

*Example:*

```
sales-data-model-template.xlsx
```

### Google Cloud Storage

Google Cloud Storage (GCS) provides durable, secure storage for objects. Use this option if your data model Excel file is stored in a GCS bucket.

#### Project ID

The ID of your Google Cloud project. You can find this in the **Google Cloud Console > Home Dashboard > Project info** panel.

- **Required** if you are using customer-managed GCS bucket.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage.

*Example:*

```
my-data-model-project-123456
```

#### Service account JSON

A JSON key file containing service account credentials with permission to access the bucket. You can create this in the **Google Cloud Console > IAM & Admin > Service accounts**.

- **Required** if you are using customer-managed GCS bucket.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage.

*Example:*

```json
{
 "type": "service_account",
 "project_id": "my-data-model-project-123456",
 "private_key_id": "abc123def456...",
 "private_key": "--BEGIN PRIVATE KEY--\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC...\n--END PRIVATE KEY--\n",
 "client_email": "data-model-import@my-data-model-project-123456.iam.gserviceaccount.com",
 "client_id": "123456789012345678901",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "token_uri": "https://oauth2.googleapis.com/token"
}
```

#### Bucket

The name of the GCS bucket containing your data model file. You can find this in the **Cloud Storage > Buckets** page.

- **Required** if you are using customer-managed GCS bucket.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage and don't know the specific bucket name.

*Example:*

```
my-company-data-model-storage
```

#### Prefix 

Optional. Directory path within the bucket where your data model file is stored. 

*Example:*

```
data-models/finance/
```

#### Object key 

The file name (including extension) of your data model file within the bucket or prefix. 

*Example:*

```
finance-data-model-template.xlsx
```

### Azure Data Lake Storage

Azure Data Lake Storage (ADLS) provides scalable data lake storage for big data analytics. Use this option if your data model Excel file is stored in an ADLS container.

#### Client ID

The application (client) ID of your Azure Active Directory app registration. You can find this in the **Azure Portal > Azure Active Directory > App registrations**.

- **Required** if you are using customer-managed Azure storage.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage.

*Example:*

```
12345678-1234-1234-1234-123456789012
```

#### Client secret

The client secret value for your Azure AD app registration. This is generated when you create a new client secret in the app registration.

- **Required** if you are using customer-managed Azure storage.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage.

*Example:*

```
abc123def456ghi789jkl012mno345pqr678stu
```

#### Tenant ID

The Azure Active Directory tenant ID. You can find this in the **Azure Portal > Azure Active Directory > Overview**.

- **Required** if you are using customer-managed Azure storage.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage.

*Example:*

```
87654321-4321-4321-4321-210987654321
```

#### Account name

The name of your Azure Data Lake Storage account. You can find this in the **Azure Portal > Storage accounts**.

- **Required** if you are using customer-managed Azure storage.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage and don't know the specific account name.

*Example:*

```
mydatamodelstorage
```

#### Container

The name of the container within your ADLS account where your data model file is stored.

- **Required** if you are using customer-managed Azure storage.
- **Keep empty** if you are using tenant-backed (Atlan-managed) storage and don't know the specific container name.

*Example:*

```
data-models
```

#### Prefix 

Optional. Directory path within the container where your data model file is stored. 

*Example:*

```
sales-models/
```

#### Object key 

The file name (including extension) of your data model file within the container or prefix. 

*Example:*

```
sales-data-model-template.xlsx
```

## Excel input 

The input file for **Data Model Ingestion** must follow the provided **Excel format**. The Data Model Ingestion app works by reading a structured Excel file that describes your data model. This file contains up to four sheets, each serving a different purpose in defining and enriching your model:

- **Objects**: defines the core building blocks: models, entities, and attributes.

- **Relations**: captures how entities are connected to each other.

- **Mappings**: links entities and attributes across tiers.

- **Implementations**: binds your model to real-world assets in Atlan (Snowflake, BigQuery, etc.). 

You can [📥 download the sample Excel template](https://docs.atlan.com/files/data-model-ingestion-package-template.xlsx) for the correct structure and example data. 

At a minimum, you must populate the **Objects** sheet. The other sheets, **Relations**, **Mappings**, and **Implementations** are optional, but recommended if you want to capture richer metadata.

If any of the optional sheets are left empty, those aspects of the model simply won't be ingested. Below are the required formats for each sheet.

### Objects Sheet

The **Objects** sheet defines the core building blocks of the data model—models, entities, and attributes. Each row in the **Objects** sheet represents either an **attribute** within an entity, or just the **entity itself** (if no attributes are defined). 

- **Model-level fields** (`Model Name`, `Model Display Name`, `Model Description`, `Model Type`) remain the same for all entities that belong to that model. 
- **Entity-level fields** (`Entity Name`, `Entity Display Name`, `Entity Description`) remain the same for all attributes of that entity. 
- **Attribute-level fields** (`Attribute Name`, `Attribute Display Name`, `Attribute Description`, `Attribute Data Type`, `Attribute Is PK`) vary for each attribute. 

**Structure:**
- Models are the top-level layer
- Each model can contain entities
- Each entity can contain attributes

**Columns:**

| Column | Description | Example |
|--------|-------------|---------|
| **Model Name*** | The technical name of the model | `SalesOverview` |
| **Model Display Name*** | The human-readable name shown in Atlan | `Sales Overview` |
| **Model Description** | A summary of what this model represents | `Comprehensive view of sales data and performance metrics` |
| **Model Type*** | Classification of the model | `CONCEPTUAL`, `LOGICAL`, or `PHYSICAL` |
| **Entity Name** | The technical name of the entity | `Account`, `Opportunity` |
| **Entity Display Name** | The human-readable name of the entity | `Account`, `Sales Opportunity` |
| **Entity Description** | A business-friendly description of what this entity represents | `Customer account information and details` |
| **Attribute Name** | The technical field/column name | `AccountNumber`, `AccountName` |
| **Attribute Display Name** | A user-friendly name for the attribute | `Account Number`, `Account Name` |
| **Attribute Description** | Describes what the attribute means | `Unique identifier for the account` |
| **Attribute Data Type** | The type of data stored (must be all caps) | `NUMBER`, `STRING`, `DATE` |
| **Attribute Is PK** | Marks whether this attribute is the Primary Key | `TRUE` or `FALSE` |

> *Example: Sales overview with accounts and opportunities — see full content on the documentation site.*

### Relations

The **Relations** sheet defines how entities connect to each other within models. Each row in the **Relations** sheet defines a relationship between two entities. 

- **Model-level** fields (`Model Type`, `Source Model`, `Destination Model`) often stay the same for multiple rows. 
- **Entity-level** fields (`Source Entity`, `Destination Entity`) remain the same for all attributes or sub-relationships under that entity pairing. 
- **Attribute-level** fields (`Relation Source Attribute`, `Relation Destination Attribute`) are optional—fill them only if you want to define relationships at the attribute level instead of the entity level. 
- **Cardinality fields** (`Source Cardinality`, `Source Min Cardinality`, `Destination Cardinality`, `Destination Min Cardinality`) define both **multiplicity** (one vs many) and **optionality** (mandatory vs optional). 

**How to interpret cardinality values:**

- `1` → one 
- `*` → many 
- **Min Cardinality**: 
 - `0` → optional (may be absent) 
 - `1` → mandatory (must exist) 

For example: 
- **1-to-Many Optional**: `Source (1, min=1) → Destination (*, min=0)` 
 - Every source must have at least one destination. 
 - A destination may or may not be linked to a source. 
- **1-to-1 Mandatory**: `Source (1, min=1) ↔ Destination (1, min=1)` 
 - Strict one-to-one relationship required on both sides. 
- **Many-to-Many**: `Source (*, min=0/1) ↔ Destination (*, min=0/1)` 
 - Both sides can have many, with optional or mandatory minimums. 

These fields directly influence the ER diagram visualization using **crow’s foot notation**: 
- Without Min Cardinalities → simplified symbols (`|` for one, `>` for many). 
- With Min Cardinalities → full notation (`o` for optional, `||` for mandatory). 

**Columns:**

| Column | Description | Example |
|--------|-------------|---------|
| **Model Type*** | Specifies the model type | `CONCEPTUAL`, `LOGICAL`, or `PHYSICAL` |
| **Source Model*** | The model that contains the starting entity | `SalesOverview` |
| **Source Entity*** | The entity from which the relationship originates | `Account` |
| **Destination Model*** | The model containing the related entity | `SalesOverview` |
| **Destination Entity*** | The entity to which the relationship points | `Opportunity` |
| **Relation Type*** | Defines the type of relationship | `Association`, `Inheritance` |
| **Cardinality Nullability** | Defines the relationship's multiplicity (high-level) | `ONE-TO-MANY`, `MANY-TO-MANY` |
| **Relation Name** | A descriptive name for the relationship | `AccountHasOpportunities` |
| **Relation Source Attribute** | Attribute in the source entity used for the relationship (leave empty for entity-level relationships) | `AccountNumber` |
| **Relation Destination Attribute** | Attribute in the destination entity used for the relationship (leave empty for entity-level relationships) | `AccountNumber` |
| **Source-to-Destination Label*** | A user-friendly label describing the relationship from source to destination | `parentOf` |
| **Destination-to-Source Label*** | Reverse label describing the relationship from destination to source | `childOf` |
| **Source Cardinality** | Maximum number of destination entities related to one source | `1`, `*` |
| **Source Min Cardinality** | Minimum number of destination entities required for each source | `0`, `1` |
| **Destination Cardinality** | Maximum number of source entities related to one destination | `1`, `*` |
| **Destination Min Cardinality** | Minimum number of source entities required for each destination | `0`, `1` |

> *Example: Defining relationships between entities — see full content on the documentation site.*

### Mappings

The **Mappings** sheet defines how different layers of your data model connect with each other across the three model tiers — Conceptual, Logical, and Physical. It captures how entities (and optionally attributes) in one model correspond to entities in another model. Each row in the **Mappings** sheet links entities (and optionally attributes) across model tiers: **Conceptual ↔ Logical ↔ Physical**. 

- **Model-level fields** (`Source Model Type`, `Source Model`, `Destination Model Type`, `Destination Model`) usually stay the same across multiple rows. 
- **Entity-level fields** (`Source Entity`, `Destination Entity`) remain the same for all attributes in that mapping. 
- **Attribute-level fields** (`Source Attribute`, `Destination Attribute`) are optional. Leave them blank if the mapping is at the entity level only. 

**Supported mapping modes:**
- **Conceptual → Logical → Physical** (full hierarchy mapping)
- **Conceptual → Logical**
- **Logical → Physical**
- **Standalone** (within the same model type, mapping attributes/entities within itself)

**Columns:**

| Column | Description | Example |
|--------|-------------|---------|
| **Source Model Type** | Defines the type of the source model | `CONCEPTUAL`, `LOGICAL`, or `PHYSICAL` |
| **Source Model*** | The name of the model from which the mapping originates | `SalesOverview` |
| **Source Entity*** | The entity in the source model that you are mapping | `Account` |
| **Destination Model Type*** | Specifies the type of model being mapped to | `CONCEPTUAL`, `LOGICAL`, or `PHYSICAL` |
| **Destination Model*** | The target model name | `SalesLogical` |
| **Destination Entity*** | The target entity name in the destination model | `Customer` |
| **Source Attribute** | The attribute in the source entity (leave blank for entity-level mappings) | `AccountNumber` |
| **Destination Attribute** | The attribute in the destination entity (leave blank for entity-level mappings) | `CustomerId` |

> *Example: Mapping conceptual, logical, and physical models — see full content on the documentation site.*

### Implementations

The **Implementations** sheet connects your logical or physical entities and attributes to actual system assets in Atlan (for example, Snowflake tables and columns). 

- **Model-level fields** (`Source Model Type`, `Source Model`) remain the same for all rows within the same model. 
- **Entity-level field** (`Source Entity`) stays the same for all attributes under that entity. 
- **Source Attribute** is optional—leave it blank if you are linking at the entity level only. 
- **Destination fields** (`Destination Asset Type`, `Destination Connector Type`, `Destination Connection Name`, `Destination Asset Qualified Name`) are always required. 

:::info Supported Asset Types
Currently supported asset types are: `MaterialisedView`, `Table`, `Column`, `GCS Object`, `S3Object`, `S3 Bucket`, `GCS Bucket`, `ADLSObject`. If you need support for additional asset types, [raise a support request](https://docs.atlan.com/support/submit-request).
:::

**Columns:**

| Column | Description | Example |
|--------|-------------|---------|
| **Source Model Type*** | Type of model | `CONCEPTUAL`, `LOGICAL`, or `PHYSICAL` |
| **Source Model*** | The name of the model | `SalesLogical` |
| **Source Entity*** | The entity being mapped | `Customer` |
| **Source Attribute** | The specific attribute in that entity (leave empty for entity-level mappings) | `CustomerId` |
| **Destination Asset Type*** | Type of asset in Atlan | `Table`, `Column`, `MaterialisedView`, `GCS Object`, `S3Object`, `S3 Bucket`, `GCS Bucket`, `ADLSObject` |
| **Destination Connector Type*** | The system type | `Snowflake`, `BigQuery`, `PostgreSQL` |
| **Destination Connection Name*** | The name of the connection configured in Atlan | `production`, `staging` |
| **Destination Asset Qualified Name*** | The qualified name of the asset (excluding connection QN prefix) | `ANALYTICS/WIDE_WORLD_IMPORTERS/people_snfw` |

> *Example: Implementing logical entities in physical assets — see full content on the documentation site.*

## Connection 

The **Connection** property defines where the ingested data model is stored and how it links to assets in Atlan. A connection represents a technical integration with a data source such as Snowflake, BigQuery, or Redshift. 

There are two options for configuring the connection: 

- **Create**: This option establishes a new connection in Atlan. It's used when onboarding a new system or domain, or when the data model must remain logically separate from existing connections. The workflow collects connection details such as type, credentials, and configuration. Once created, the connection is available for reuse in future ingestion jobs.

 *Example:* Creating a dedicated `Finance_Prod` connection to hold the finance data model for the first time. 

- **Reuse**: This option links the data model to an existing connection in Atlan. It's used when enriching or extending metadata for assets that are already cataloged under a configured connection. The ingestion job attaches the new metadata to the selected connection, keeping related information grouped under one integration.

 *Example:* Reusing an existing `Snowflake_Prod` connection when ingesting a sales conceptual model that maps onto Snowflake assets already present in Atlan. 

## Connection name 

The connection name applies only when **Create** is selected and specifies the display name of the new connection. This name is used to identify the connection in the Atlan UI and across workflows.

The name must be unique and descriptive, reflecting the source system or domain being modeled (for example, `Finance_Prod` or `Sales_Snowflake`). Once the connection is created, the name is permanent and available for reuse in future ingestion workflows.

## Connection admins 

Applies only when **Create** is selected. 

Defines the list of users who administer the connection. Connection admins have full control over the connection, including configuration and management of associated workflows. 

- **Assign specific users**: Add one or more users as connection admins by specifying their Atlan usernames. 
- **Include all admins**: Select this option to grant all tenant admins administrative rights on the connection. 

*Example:* When creating a new connection named `Finance_Prod`, specify `data.platform@org.com` and `jane.doe@org.com` as connection admins, or select **Include all admins** to grant rights to all tenant administrators. 

## Connection 

The connection applies only when **Reuse** is selected and specifies the existing connection in Atlan where the data model is ingested. The ingestion process attaches the new metadata to the selected connection, keeping all related information grouped under one integration.

You must select the connection name from the list of available connections in the tenant and verify that it corresponds to the data source or domain that the ingested model is intended to enrich. Using an existing connection avoids duplication and maintains consistency across ingestion jobs. For example, when ingesting a sales conceptual model that maps to Snowflake assets already cataloged in Atlan, you can select the existing connection `Snowflake_Prod`.

## See also

- [Ingest Data Models](https://docs.atlan.com/llms/governance/data-models/data-model-ingestion/llms.txt)

---