## Manage AWS S3 assets

URL: https://docs.atlan.com/apps/connectors/storage/amazon-s3/sdk/how-tos/manage-assets

> Manage Amazon S3 assets programmatically in Atlan using the Python SDK (pyatlan). Create, update, and delete S3 connections, buckets, and objects via the SDK.

# ![S3](http://assets.atlan.com/assets/s3-logo.svg) Manage AWS S3 assets

Operations on S3 assets (connections, buckets, objects).

In general, these should be:

- Created in top-down order (connection, then bucket, then object)
- Deleted in bottom-up order (objects, then buckets, then connections)[^1]

```mermaid
erDiagram
 Connection ||--o{ S3Bucket : contains
 S3Bucket ||--o{ S3Object : contains
```

## Asset structure

### Connection

An AWS S3 [connection](https://docs.atlan.com/llms/platform/types/connection/llms.txt) requires a `name` and `qualifiedName`. For creation, specific settings are also required to distinguish it as an AWS S3 connection rather than another type of connection. In addition, *at least one* of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.

### Java

```java showLineNumbers title="Create an S3 connection"
String adminRoleGuid = client.getRoleCache().getIdForName("$admin"); // (1)
Connection connection = Connection.creator( // (2)
 "aws-s3-connection", // (3)
 AtlanConnectorType.S3, // (4)
 List.of(adminRoleGuid), // (5)
 List.of("group2"), // (6)
 List.of("jsmith")) // (7)
 .build();
AssetMutationResponse response = connection.save(client); // (8)
String connectionQualifiedName = response.getCreatedAssets().get(0).getQualifiedName(); // (9)
```

1. Retrieve the GUID for the admin role, to use later for defining the roles that can administer the connection.
2. Build up the minimum request to create a connection.
3. Provide a human-readable name for your connection, such as `production` or `development`.
4. Set the type of connection to S3.
5. List the workspace roles that should be able to administer the connection (or null if none). All users with that workspace role (current and future) will be administrators of the connection. Note that the values here need to be the GUIDs of the workspace roles. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
6. List the group names that can administer this connection (or null if none). All users within that group (current and future) will be administrators of the connection. Note that the values here are the names of the groups. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
7. List the user names that can administer this connection (or null if none). Note that the values here are the usernames of the users. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
8. Actually call Atlan to create the connection. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.
9. Retrieve the qualifiedName for use in subsequent creation calls. (You'd probably want to do some null checking first.)

### Python

```python showLineNumbers title="Create an S3 connection"
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import Connection, S3Bucket, S3Object
from pyatlan.model.enums import AtlanConnectorType

client = AtlanClient()
admin_role_guid = client.role_cache.get_id_for_name("$admin") # (1)
connection = Connection.creator( # (2)
 client=client, # (3)
 name="aws-s3-connection", # (4)
 connector_type=AtlanConnectorType.S3, # (5)
 admin_roles=[admin_role_guid], # (6)
 admin_groups=["group2"], # (7)
 admin_users=["jsmith"], # (8)
)
response = client.asset.save(connection) # (9)
connection_qualified_name = response.assets_created(asset_type=Connection)[0].qualified_name # (10)
```

1. Retrieve the GUID for the admin role, to use later for defining the roles that can administer the connection.
2. Build up the minimum request to create a connection.
3. You must provide a client instance.
4. Provide a human-readable name for your connection, such as `production` or `development`.
5. Set the type of connection to S3.
6. List the workspace roles that should be able to administer the connection (if any, defaults to `None`). All users with that workspace role (current and future) will be administrators of the connection. Note that the values here need to be the GUIDs of the workspace roles. *At least one of `admin_roles`, `admin_groups`, or `admin_users` must be provided.*
7. List the group names that can administer this connection (if any, defaults to `None`). All users within that group (current and future) will be administrators of the connection. Note that the values here are the names of the groups. *At least one of `admin_roles`, `admin_groups`, or `admin_users` must be provided.*
8. List the user names that can administer this connection (if any, defaults to `None`). Note that the values here are the usernames of the users. *At least one of `admin_roles`, `admin_groups`, or `admin_users` must be provided.*
9. Actually call Atlan to create the connection.
10. Retrieve the qualified_name for use in subsequent creation calls. (You'd probably want to do some other checks first.)

### Kotlin

```kotlin showLineNumbers title="Create an S3 connection"
val adminRoleGuid = client.roleCache.getIdForName("\$admin"); // (1)
val connection = Connection.creator( // (2)
 "aws-s3-connection", // (3)
 AtlanConnectorType.S3, // (4)
 listOf(adminRoleGuid), // (5)
 listOf("group2"), // (6)
 listOf("jsmith")) // (7)
 .build()
val response = connection.save(client) // (8)
val connectionQualifiedName = response.createdAssets[0].qualifiedName // (9)
```

1. Retrieve the GUID for the admin role, to use later for defining the roles that can administer the connection.
2. Build up the minimum request to create a connection.
3. Provide a human-readable name for your connection, such as `production` or `development`.
4. Set the type of connection to S3.
5. List the workspace roles that should be able to administer the connection (or null if none). All users with that workspace role (current and future) will be administrators of the connection. Note that the values here need to be the GUIDs of the workspace roles. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
6. List the group names that can administer this connection (or null if none). All users within that group (current and future) will be administrators of the connection. Note that the values here are the names of the groups. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
7. List the user names that can administer this connection (or null if none). Note that the values here are the usernames of the users. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
8. Actually call Atlan to create the connection. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.
9. Retrieve the qualifiedName for use in subsequent creation calls. (You'd probably want to do some null checking first.)

### Raw REST API

```json showLineNumbers title="POST /api/meta/entity/bulk"
{
 "entities": [
 {
 "typeName": "Connection", // (1)
 "attributes": {
 "name": "aws-s3-connection", // (2)
 "connectorName": "s3", // (3)
 "qualifiedName": "default/s3/123456789", // (4)
 "category": "ObjectStore", // (5)
 "adminRoles": [ // (6)
 "e7ae0295-c60a-469a-bd2c-fb903943aa02"
 ],
 "adminGroups": [ // (7)
 "group2"
 ],
 "adminUsers": [ // (8)
 "jsmith"
 ]
 }
 }
 ]
}
```

1. The `typeName` must be exactly `Connection`.
2. Human-readable name for your connection, such as `production` or `development`.
3. The `connectorName` must be exactly `s3`.
4. The `qualifiedName` should follow the pattern: `default/s3/<epoch>`, where `<epoch>` is the time in milliseconds at which the connection is being created.
5. The `category` must be `ObjectStore`.
6. List any workspace roles that can administer this connection. All users with that workspace role (current and future) will be administrators of the connection. Note that the values here need to be the GUIDs of the workspace roles. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
7. List any groups that can administer this connection. All users within that group (current and future) will be administrators of the connection. Note that the values here are the names of the groups. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*
8. List any users that can administer this connection. Note that the values here are the usernames of the users. *At least one of `adminRoles`, `adminGroups`, or `adminUsers` must be provided.*

:::warning[Access policies]
Atlan creates the policies that grant access to a connection, including the ability to retrieve the connection and to create assets within it, asynchronously. It can take several seconds (even up to approximately 30 seconds) before these are in place after creating the connection.

You may therefore need to wait before you'll be able to create the assets below within the connection.

To confirm access, [retrieve the connection](https://docs.atlan.com/llms/platform/python/retrieve-asset/llms.txt) after it has been created. The SDKs' retry loops will automatically retry until the connection can be successfully retrieved. At that point, your API token has permission to create the other assets.

Note: if you are *reusing* an existing connection rather than creating one via your API token, you must give your API token a persona that has access to that connection. Otherwise all attempts to create, read, update, or delete assets within that connection will fail due to a lack of permissions.
:::

### S3Bucket

An AWS S3 [bucket](https://docs.atlan.com/llms/platform/types/s3bucket/llms.txt) requires both a `name` and a `qualifiedName`. During creation, you also need to specify the `connectionQualifiedName` of the connection associated with the bucket, and optionally provide a unique `awsArn`.

### Java

```java title="Create an S3 bucket"
S3Bucket s3Bucket = S3Bucket.creator( // (1)
		"mybucket", // (2)
		connectionQualifiedName, // (3)
		"arn:aws:s3:::mybucket") // (4)
 .s3ObjectCount(10) // (5)
 .build();
AssetMutationResponse response = s3Bucket.save(client); // (6)
s3Bucket = response.getResult(s3Bucket); // (7)
```

1. Build up the minimum request to create a bucket.
2. Provide a human-readable name for your bucket.
3. Provide the `qualifiedName` of the connection for this bucket.
4. (Optional) If `awsArn` is provided, it will be used to construct the `qualifiedName` for the bucket; otherwise, the `name` of the bucket will be used.
5. (Optional) To make sure the UI displays the correct count of `S3Object`'s, set the `s3ObjectCount` directly on the `S3Bucket` instance.
6. Actually call Atlan to create the bucket. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.
7. Retrieve the created bucket for use in subsequent creation calls. (You'd probably want to do some null checking first.)

### Python

```python title="Create an S3 bucket"
s3bucket = S3Bucket.creator( # (1)
 name="mybucket", # (2)
 connection_qualified_name=connection_qualified_name, # (3)
 aws_arn="arn:aws:s3:::mybucket" # (4)
)
s3bucket.s3_object_count = 10 # (5)
response = client.asset.save(s3bucket) # (6)
bucket_qualified_name = response.assets_created(asset_type=S3Bucket)[0].qualified_name # (7)
```

1. Build up the minimum request to create a bucket.
2. Provide a human-readable name for your bucket.
3. Provide the `qualified_name` of the connection for this bucket.
4. (Optional) If `aws_arn` is provided, it will be used to construct the `qualified_name`
for the bucket; otherwise, the `name` of the bucket will be used.
5. (Optional) To make sure the UI displays the correct count of `S3Object`'s,
set the `s3_object_count` directly on the `S3Bucket` instance.
6. Actually call Atlan to create the bucket.
7. Retrieve the `qualified_name` for use in subsequent creation calls.
(You'd probably want to do some checks first.)

### Kotlin

```kotlin title="Create an S3 bucket"
var s3Bucket = S3Bucket.creator( // (1)
 "mybucket", // (2)
 connectionQualifiedName, // (3)
 "arn:aws:s3:::mybucket") // (4)
 .s3ObjectCount(10) // (5)
 .build()
val response = s3Bucket.save(client) // (6)
s3Bucket = response.getResult(s3Bucket) // (7)
```

1. Build up the minimum request to create a bucket.
2. Provide a human-readable name for your bucket.
3. Provide the `qualifiedName` of the connection for this bucket.
4. (Optional) If `awsArn` is provided, it will be used to construct the `qualifiedName` for the bucket; otherwise, the `name` of the bucket will be used.
5. (Optional) To make sure the UI displays the correct count of `S3Object`'s, set the `s3ObjectCount` directly on the `S3Bucket` instance.
6. Actually call Atlan to create the bucket. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.
7. Retrieve the created bucket for use in subsequent creation calls. (You'd probably want to do some null checking first.)

### Raw REST API

```json showLineNumbers title="POST /api/meta/entity/bulk"
{
 "entities": [
 {
 "typeName": "S3Bucket", // (1)
 "attributes": {
 "name": "mybucket", // (2)
 "awsArn": "arn:aws:s3:::mybucket", // (3)
 "qualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket", // (4)
 "connectionQualifiedName": "default/s3/123456789", // (5)
 "connectorName": "s3" // (6)
 }
 }
 ]
}
```

1. The `typeName` must be exactly `S3Bucket`.
2. Human-readable name for your bucket.
3. The `awsArn` should be the unique ARN from AWS for this bucket.
4. The `qualifiedName` should follow the pattern: `default/s3/<epoch>/<awsArn>`, where `default/s3/<epoch>` is the qualifiedName of the connection for this bucket and `<awsArn>` is the unique ARN for this bucket.
5. The `connectionQualifiedName` must be the exact qualifiedName of the connection for this bucket.
6. The `connectorName` must be exactly `s3`.

### S3Object

An AWS S3 [object](https://docs.atlan.com/llms/platform/types/s3object/llms.txt) requires a `name` and a `qualifiedName`. For creation, you also need to specify the `connectionQualifiedName` of the connection for the object, and a unique `awsArn` or `prefix`. You should also specify the `bucket` the object is in, along with its `s3BucketName` and `s3BucketQualifiedName`.

#### By AWS ARN

Create an S3 object using AWS ARN:

### Java

```java title="Create an S3 object using AWS ARN"
S3Object s3Object = S3Object.creator( // (1)
		"myobject.csv", // (2)
		s3Bucket, // (3)
		"arn:aws:s3:::mybucket/prefix/myobject.csv") // (4)
 .build();
AssetMutationResponse response = s3Object.save(client); // (5)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the bucket in which this object should be created.
4. Provide the unique ARN from AWS for this object.
5. Actually call Atlan to create the object. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.

### Python

```python title="Create an S3 object using AWS ARN"
s3object = S3Object.creator( # (1)
 name="myobject.csv", # (2)
 connection_qualified_name=connection_qualified_name, # (3)
 aws_arn="arn:aws:s3:::mybucket/prefix/myobject.csv", # (4)
 s3_bucket_name=bucket_name, # (5)
 s3_bucket_qualified_name=bucket_qualified_name, # (6)
)
response = client.asset.save(s3object) # (7)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the `qualified_name` of the connection for this object.
4. Provide the unique ARN from AWS for this object.
5. Provide the `name` of the bucket this object should be created within.
6. Provide the `qualified_name` of the bucket this object should be created within.
7. Actually call Atlan to create the object.

### Kotlin

```kotlin title="Create an S3 object using AWS ARN"
var s3Object = S3Object.creator( // (1)
 "myobject.csv", // (2)
 s3Bucket, // (3)
 "arn:aws:s3:::mybucket/prefix/myobject.csv") // (4)
 .build()
val response = s3Object.save(client) // (5)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the bucket in which this object should be created.
4. Provide the unique ARN from AWS for this object.
5. Actually call Atlan to create the object. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.

### Raw REST API

```json showLineNumbers title="POST /api/meta/entity/bulk"
{
 "entities": [
 {
 "typeName": "S3Object", // (1)
 "attributes": {
 "name": "myobject.csv", // (2)
 "awsArn": "arn:aws:s3:::mybucket/prefix/myobject.csv", // (3)
 "qualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket/prefix/myobject.csv", // (4)
 "connectionQualifiedName": "default/s3/123456789", // (5)
 "connectorName": "s3", // (6)
 "bucket": { // (7)
 "typeName": "S3Bucket", // (8)
 "uniqueAttributes": { // (9)
 "qualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket"
 }
 },
 "s3BucketName": "mybucket", // (10)
 "s3BucketQualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket" // (11)
 }
 }
 ]
}
```

1. The `typeName` must be exactly `S3Object`.
2. Human-readable name for your object.
3. The `awsArn` should be the unique ARN from AWS for this object.
4. The `qualifiedName` should follow the pattern: `default/s3/<epoch>/<awsArn>`,
where `default/s3/<epoch>` is the `qualifiedName` of the connection for this
object and `<awsArn>` is the unique ARN for this object.
5. The `connectionQualifiedName` must be the exact qualifiedName of the connection for this object.
6. The `connectorName` must be exactly `s3`.
7. The bucket in which this object exists is embedded in the `bucket` attribute.
8. The `typeName` for this embedded reference must be `S3Bucket`.
9. To complete the reference, you must include a `uniqueAttributes` object with the qualifiedName of the bucket. Note: the bucket must already exist in Atlan before creating the object.
10. The `s3BucketName` should be the human-readable name of the bucket.
11. The `s3BucketQualifiedName` should be the qualifiedName of the bucket.

#### By prefix

Create an S3 object using prefix:

### Java

```java title="Create an S3 object using prefix"
S3Object s3Object = S3Object.creatorWithPrefix( // (1)
		"myobject.csv", // (2)
		s3Bucket, // (3)
		"/some/folder/structure") // (4)
 .build();
AssetMutationResponse response = s3Object.save(client); // (5)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the bucket in which this object should be created.
4. Provide the folder path where the object is located within the bucket.
5. Actually call Atlan to create the object. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.

### Python

```python title="Create an S3 object using prefix"
s3object = S3Object.creator_with_prefix( # (1)
 name="myobject.csv", # (2)
 connection_qualified_name=connection_qualified_name, # (3)
 prefix="/some/folder/structure", # (4)
 s3_bucket_name=bucket_name, # (5)
 s3_bucket_qualified_name=bucket_qualified_name, # (6)
)
response = client.asset.save(s3object) # (7)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the `qualified_name` of the connection for this object.
4. Provide the folder path where the object is located within the bucket.
5. Provide the `name` of the bucket this object should be created within.
6. Provide the `qualified_name` of the bucket this object should be created within.
7. Actually call Atlan to create the object.

### Kotlin

```kotlin title="Create an S3 object using prefix"
var s3Object = S3Object.creatorWithPrefix( // (1)
 "myobject.csv", // (2)
 s3Bucket, // (3)
 "/some/folder/structure") // (4)
 .build()
val response = s3Object.save(client) // (5)
```

1. Build up the minimum request to create an object.
2. Provide a human-readable name for your object.
3. Provide the bucket in which this object should be created.
4. Provide the folder path where the object is located within the bucket.
5. Actually call Atlan to create the object. Because this operation will persist the asset in Atlan, you must [provide it an `AtlanClient`](https://docs.atlan.com/llms/platform/python/set-up-sdk/llms.txt) through which to connect to the tenant.

### Raw REST API

```json showLineNumbers title="POST /api/meta/entity/bulk"
{
 "entities": [
 {
 "typeName": "S3Object", // (1)
 "attributes": {
 "name": "myobject.csv", // (2)
 "s3ObjectKey": "/some/folder/structure/myobject.csv", // (3)
 "qualifiedName": "default/s3/123456789/mybucket//some/folder/structure/myobject.csv",
 // (4)
 "connectionQualifiedName": "default/s3/123456789", // (5)
 "connectorName": "s3", // (6)
 "bucket": { // (7)
 "typeName": "S3Bucket", // (8)
 "uniqueAttributes": { // (9)
 "qualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket"
 }
 },
 "s3BucketName": "mybucket", // (10)
 "s3BucketQualifiedName": "default/s3/123456789/arn:aws:s3:::mybucket" // (11)
 }
 }
 ]
}
```

1. The `typeName` must be exactly `S3Object`.
2. Human-readable name for your object.
3. Provide the folder path where the object is located within the bucket.
4. The `qualifiedName` should follow the pattern: `default/s3/<epoch>/&lt;bucket-name&gt;/<prefix>/<name>`,
where `default/s3/<epoch>` is the `qualifiedName` of the connection for this object,
and `<prefix>/<name>` is the folder path where this object is located within the bucket.
5. The `connectionQualifiedName` must be the exact qualifiedName of the connection for this object.
6. The `connectorName` must be exactly `s3`.
7. The bucket in which this object exists is embedded in the `bucket` attribute.
8. The `typeName` for this embedded reference must be `S3Bucket`.
9. To complete the reference, you must include a `uniqueAttributes` object with the qualifiedName of the bucket. Note: the bucket must already exist in Atlan before creating the object.
10. The `s3BucketName` should be the human-readable name of the bucket.
11. The `s3BucketQualifiedName` should be the qualifiedName of the bucket.

## Available relationships

Every level of the object store structure is an `Asset`, and can therefore be related to the following other assets.

[^1]: Although if you want to delete everything in a connection, your better avenue is the packaged [connection delete utility](https://ask.atlan.com/hc/en-us/articles/6755306791697) in the UI.

---