Google Big Query

Steps to integrate your Google Big Query storage with Atlan

Atlan natively supports Google Big Query, which allows you to seamlessly integrate your Big Query with your Atlan workspace.

๐Ÿ’ญ TL;DR

  1. Select the source, aka Big Query ๐Ÿ˜‰

  2. Provide your credentials โœ๏ธ

  3. Set up your configuration ๐Ÿ—„๏ธ

  4. Schedule automatic updates ๐Ÿ•‘

๐Ÿ“œ Prerequisites for a Big Query integration

  • Service Accounts: Service accounts are special Google accounts that can be used by applications to access Google APIs programmatically via OAuth 2.0. A service account uses an OAuth 2.0 flow that does not require additional authorization. Instead, it uses a key file that only your application can access

  • Project ID: The project ID is a unique identifier for a Google Cloud project.

๐ŸŒŸ Pro Tip: If you don't have this information handy, reach out to your cloud or data warehouse administrator to get these details before you get started!

๐Ÿ”‘ Big Query user permissions

Make sure you have permission to grant access to resources. For more information, visit "Granting, changing, and revoking access to resources".

Create and download your service account JSON file with the following permissions: roles/bigquery.dataViewer

For detailed steps on creating a service account, visit "Creating and managing service account keys".

๐Ÿš€ A step-by-step guide to integrate Big Query with Atlan

STEP 1: Select the source

  1. Log into your Atlan workspace.

  2. On the home screen, click on the "New Integration" button in the top right corner. You will see a dialogue box with the list of sources available on your workspace.

  3. Select "Big Query" from the list of options, and click on "Next".

Select Big Query to add new integrations

STEP 2: Provide credentials

  1. You will see an option to either select a preconfigured credential from the drop-down menu or to create a credential. To set up a new connection, click on the "Create Credential" button.

  2. You will be required to fill in your Big Query credentials. Below is an example of the credentials required:

    {
    "type": "service_account",
    "project_id": "mybigproject",
    "private_key_id": "0adfdf1",
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEghoQ=\n-----END PRIVATE KEY-----\n",
    "client_email": "mybigproj@mybigorg.iam.gserviceaccount.com",
    "client_id": "12345",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/bq-test1%40atlanhq.iam.gserviceaccount.com"
    }
  3. Once you have filled in the details, click on "Next".

Big Query creds

STEP 3: Set up your configuration

  1. Specify the project ID you want to crawl.

  2. Select the data sets and tables you want to crawl. Check "Include All" to include all data sets and tables.

  3. Choose whether to run the crawler once, or schedule it for a daily, weekly, or monthly run. You will be asked to specify the time zone for the run.

  4. Click on "Create". Your connection is now created.

Congratulations! You have now integrated Atlan with your Big Query ๐ŸŽ‰

๐Ÿ Monitoring your Big Query integration

Once the integration setup is completed, you will be redirected to the "Monitor" tab for your Big Query asset, where you can monitor its progress.