Skip to main content

Object storage for apps

This reference provides complete configuration details for connecting to cloud object storage systems in Atlan apps. Many apps support importing files from cloud object stores, including Asset Import, Relational Assets Builder, Lineage Builder, and Data Model Ingestion. This guide covers all common configuration fields and authentication methods across supported providers.

Prefix (path)​

This property specifies the directory or path within your selected cloud object store where the file is located.

  • Optional: Can be left blank to search from the root of the bucket or container
  • With prefix: Only files under the specified path are processed
  • Without prefix: System searches from the root of the storage location
  • Format: Use forward slashes (/) as path separators
  • Trailing slash: Not required, Atlan appends automatically if missing
  • Path combination: Prefix + Object key = complete file location (within the bucket or container)

Example: If your file is stored in a folder called imports inside the data directory of your bucket, set the prefix to:

data/imports

Example: For nested paths:

production/data-models/2024

Object key (filename)​

This property specifies the exact file to import from your chosen cloud object store. The value entered here is combined with the optional Prefix (path) to form the complete location of the file.

  • Required: Must include the file name and extension
  • Single file: Only one file can be provided per configuration
  • Multiple files: Requires separate workflow runs for each file, even if they're stored in the same prefix or folder
  • File extension: Must include the file extension (for example, .csv, .xlsx, .json)
  • Path combination: Object key + prefix = complete file location (within the bucket or container)

Example: If your file is stored under a folder called data/imports in your object store, you can configure:

Prefix:

data/imports

Object key:

customer-data.csv

Complete path: {{bucket}}/data/imports/customer-data.csv

Example: For files at the root level:

Prefix: (leave blank)

Object key:

metadata-export.xlsx

Complete path: {{bucket}}/metadata-export.xlsx

Cloud object store​

This property specifies which cloud provider to use for accessing files from object storage. Supported providers are Amazon S3, Google Cloud Storage (GCS), and Azure Data Lake Storage (ADLS). When this option is selected, additional storage-specific properties such as bucket, project ID, or container become available.

Amazon S3 enables you to store and retrieve objects at scale. You can use this option when your file is stored in an S3 bucket.

AWS access key​

The access key for your AWS account. You can find this in the AWS Management Console > IAM > Users > Security credentials tab.

  • Required if you are using access/secret key authentication
  • Keep empty if you are using tenant-backed, cross-account, or role-based authentication

Example:

AKIAIOSFODNN7EXAMPLE

AWS secret key​

The secret key that pairs with your access key. This is generated when you create an access key in IAM. You must download it at creation time or rotate and generate a new one if lost. The secret key is only shown once when created, so store it securely. For better security, consider using role-based authentication instead of access keys.

  • Required if you are using access/secret key authentication
  • Keep empty if you are using tenant-backed, cross-account, or role-based authentication

Example:

wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

AWS role ARN​

The ARN of the AWS role to use to access S3. You must set this up separately in AWS, and grant permissions for Atlan to assume this role. Role-based authentication is more secure than access keys because it doesn't require storing long-term credentials. The role must have a trust policy that lets Atlan assume it.

  • Required if you are using role-based authentication method
  • Keep empty if you are using tenant-backed, cross-account, or access/secret key based authentication

Example:

arn:aws:iam::123456789012:role/atlan-s3-access-role
Role-based authentication

When using role-based authentication, the role must have the necessary S3 permissions (for example, s3:GetObject and s3:ListBucket) for the bucket and prefix you're accessing.

Region​

The AWS region in which your bucket is located (for example, us-east-1). You can find this in the S3 service dashboard when selecting your bucket.

  • Required if you are using access/secret key authentication
  • Keep empty in all other scenarios, where the region is inferred from the tenant or role

Example:

ap-southeast-1

Example: Common regions:

us-east-1
us-west-2
eu-west-1
ap-southeast-1

Bucket​

The name of the S3 bucket that contains your file. The bucket name is listed in the S3 service dashboard.

  • Keep empty to use the tenant-backed object store's bucket (if available)
  • Required in all other scenarios (cross-account, access/secret keys, and role-based)

Example:

my-company-data-imports
Bucket naming

S3 bucket names must be globally unique and follow AWS naming conventions (lowercase letters, numbers, hyphens, and periods).

See also​