πŸ“œ
Our Manifesto
🧰
Backup & Disaster Recovery
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Customer Success & Support
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Community
Google Big Query
Steps to integrate your Google Big Query storage with Atlan
Atlan natively supports Google Big Query, which allows you to seamlessly integrate your Big Query with your Atlan workspace.

πŸ’­ TL;DR

  1. 1.
    Select the source, aka Big Query πŸ˜‰
  2. 2.
    Provide your credentials ✍️
  3. 3.
    Set up your configuration πŸ—„οΈ
  4. 4.
    Schedule automatic updates πŸ•‘

πŸ“œ Prerequisites for a Big Query integration

  • Service Accounts: Service accounts are special Google accounts that can be used by applications to access Google APIs programmatically via OAuth 2.0. A service account uses an OAuth 2.0 flow that does not require additional authorization. Instead, it uses a key file that only your application can access.
  • Project ID: The project ID is a unique identifier for a Google Cloud project.
🌟 Pro Tip: If you don't have this information handy, reach out to your cloud or data warehouse administrator to get these details before you get started!

πŸ”‘ Big Query user permissions

πŸ‘€ Note: Make sure you have permissions to grant access to resources. For more information, visit "Granting, changing, and revoking access to resources".
  1. 1.
    Create a service account. For detailed steps on creating a service account, visit "Creating and managing service account keys".
  2. 2.
    Add a role to the service account, based on whether you need to just crawl or crawl and query your metadata. More information below.

Option 1: For metadata crawling only

You will need to create a custom role with the below permissions. For detailed steps, visit "Creating a custom role".
1
bigquery.datasets.get
2
bigquery.datasets.getIamPolicy
3
bigquery.jobs.create
4
bigquery.models.getMetadata
5
bigquery.models.list
6
bigquery.routines.get
7
bigquery.routines.list
8
bigquery.tables.get
9
bigquery.tables.getIamPolicy
10
bigquery.tables.list
11
resourcemanager.projects.get
Copied!
These are basically the permissions in the role roles/bigquery.metadataViewer with an extra bigquery.jobs.create permission. Atlan users will not be able to query any tables with these permissions.

Option 2: For metadata crawling, data preview, querying, and data profiling

Attach the role roles/bigquery.dataViewer and the permissions below to the service account.
1
bigquery.datasets.get
2
bigquery.jobs.create
3
bigquery.jobs.get
4
bigquery.jobs.list
5
bigquery.jobs.listAll
6
bigquery.jobs.update
7
bigquery.readsessions.create
8
bigquery.readsessions.getData
9
bigquery.readsessions.update
10
bigquery.tables.create
11
bigquery.tables.get
12
bigquery.tables.getData
13
bigquery.tables.list
Copied!

πŸš€ A step-by-step guide to integrate Big Query with Atlan

STEP 1: Select the source

  1. 1.
    Log into your Atlan workspace.
  2. 2.
    On the home screen, click on the "New Integration" button on the top right corner. You will see a dialogue box with the list of sources available on your workspace.
  3. 3.
    Select "Big Query" from the list of options, and click on "Next".

STEP 2: Provide credentials

  1. 1.
    You will see an option to either select a preconfigured credential from the drop-down menu or to create a credential. To set up a new connection, click on the "Create Credential" button.
  2. 2.
    You will be required to fill in your Big Query credentials. Below is an example of the credentials required:
Service Account:
1
{
2
"type": "service_account",
3
"project_id": "mybigproject",
4
"private_key_id": "0adfdf1",
5
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEghoQ=\n-----END PRIVATE KEY-----\n",
6
"client_email": "[email protected]",
7
"client_id": "12345",
8
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
9
"token_uri": "https://oauth2.googleapis.com/token",
10
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
11
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/bq-test1%40atlanhq.iam.gserviceaccount.com"
12
}
Copied!
  1. 1.
    Once you have filled in the details, click on "Next".
Credentials required to crawl the metadata from Big Query

STEP 3: Set up your configuration

  1. 1.
    Specify the Project ID you want to crawl.
  2. 2.
    Select the data sets and tables you want to crawl. Check "Include All" to include all data sets and tables.
  3. 3.
    Choose whether to run the crawler once or schedule it for a daily, weekly, or monthly run. You will be asked to specify the time zone for the run.
  4. 4.
    Click on "Create". Your connection is now created.
Congratulations! You have now integrated Atlan with your Big Query πŸŽ‰

🏁 Monitoring your Big Query integration

Once the integration setup is completed, you will be redirected to the "Monitor" tab for your Big Query asset, where you can monitor its progress.
Last modified 1mo ago