Skip to main content

Set up Google Cloud Storage

This guide walks you through setting up Google Cloud Storage (GCS) to enable secure data ingestion from your GCS buckets.

This guide walks you through setting up Google Cloud Storage (GCS) to enable secure data ingestion from your GCS buckets. The connector catalogs GCS buckets and objects only.

Prerequisites​

  • GCS bucket containing the data you want to ingest
  • Appropriate permissions to create service accounts and manage IAM roles

Permissions required​

Make sure you (or an administrator) can assign the following IAM roles to the service account that the connector uses:

  • Storage Bucket Viewer (roles/storage.bucketViewer)
  • Storage Object Viewer (roles/storage.objectViewer)

You also need permission to create a service account and generate its key.

Create a service account​

  1. Select your project from the project dropdown. Creating a dedicated service account avoids using personal credentials and lets you manage access centrally.

  2. In the left navigation menu, go to IAM & Admin > Service accounts.

  3. Select Create service account.

  4. Enter a name for your service account (for example, atlan-gcs-connector).

  5. Add an optional description.

  6. Select Create and continue.

Assign roles and permissions​

Add the following roles to your service account: These roles grant read-only access so the connector can discover buckets and objects without modifying data.

  • Storage Bucket Viewer: Lets you read bucket details (storage.buckets.list).
  • Storage Object Viewer: Lets you list objects and read object metadata (storage.objects.list).
  1. Select Done.

Generate a service account key​

  1. In the left navigation menu, go to IAM & Admin > Service accounts. The JSON key file is used by the connector to authenticate to GCP programmatically.

  2. Select Create key.

  3. Download and store the key file securely.

  4. Select JSON as the key type.

  5. Select Create.

  6. Download the JSON file and store it securely.

Configure bucket permissions​

  1. Navigate to Cloud Storage. Grant the service account access to every bucket you want Atlan to crawl.

  2. Select your bucket.

  3. Go to the Permissions tab.

  4. Select Add principal.

  5. Enter your service account email (for example, [email protected]).

  6. Assign the Storage Object Viewer role.

  7. Select Save.

Need help​

If you run into issues during the GCS setup process:

Next steps​

  • Crawl GCS assets: Follow this guide to configure the crawler workflow and ingest metadata from your GCS buckets.