Crawl MySQL

Once you have configured the MySQL user permissions, you can establish a connection between Atlan and MySQL. (If you are also using a private network for MySQL, set that up first.)

To crawl metadata from MySQL, review the order of operations and then complete the following steps.

Create crawler workflow

Follow these steps to create a MySQL crawler workflow:

1. In the top right of any screen, navigate to New and then click New Workflow.
2. From the list of packages, select MySQL Assets and click on Setup Workflow.

Configure extraction

Select your extraction method and provide the connection details.

Direct

In Direct extraction, Atlan connects to your database and crawls metadata directly.

1. Enter your MySQL connection details:

   • Host Name: Enter the host for your MySQL instance.
   • Port: Enter the port number of your MySQL instance (default is 3306).

2. Choose an authentication method you configured when setting up the MySQL user.

Basic authentication

Use standard database credentials created in your MySQL instance.

• Username: Enter the username you configured in MySQL.
• Password: Enter the password for the specified user.

IAM User authentication

Authenticate using AWS IAM user credentials for RDS or Aurora MySQL.

• AWS Access Key: Enter the AWS access key ID for the IAM user.
• AWS Secret Key: Enter the AWS secret access key for the IAM user.
• Username: Enter the database username you configured in MySQL.

IAM Role authentication

Authenticate using AWS IAM role for RDS or Aurora MySQL.

• AWS Role ARN: Enter the ARN of the IAM role you created.
• Username: Enter the database username you configured in MySQL.
• AWS External ID (optional): Enter the external ID only if you have configured an external ID in the role definition.

3. Click Test Authentication to confirm connectivity to MySQL using these details.

info

💪 Did you know? If you get an Error: 1129: Host ... is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts', ask your database admin to run the FLUSH HOSTS; command in the RDS instance, and then try again.

4. When successful, at the bottom of the screen click Next.

Offline

In Offline extraction, you use Atlan's metadata-extractor tool to extract metadata from MySQL and store it in S3.

1. Complete the offline extraction setup to extract and upload metadata files to S3.
2. For Bucket name, enter the name of your S3 bucket or Atlan's bucket.
3. For Bucket prefix, enter the S3 prefix under which all the metadata files exist. These include databases.json, columns-<database>.json, and similar files.
4. For Bucket region (optional), enter the name of the S3 region if your bucket is in a specific region.
5. Click Next at the bottom of the screen.

Agent

In Agent extraction, Self-Deployed Runtime executes metadata extraction within your organization's environment.

1. Install Self-Deployed Runtime if you haven't already:
   • Install via Docker Compose
   • Install on Kubernetes
2. Select the Agent tab.
3. Store sensitive information in the secret store configured with the Self-Deployed Runtime and reference the secrets in the corresponding fields. For more information, see Secret management.
4. For details on individual fields, refer to the Direct extraction tab.
5. Click Next after completing the configuration.

Configure connection

To complete the MySQL connection configuration:

1. Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics.

2. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins.

   warning

   If you don't specify any user or group, nobody can manage the connection - not even admins.

3. (Optional) To prevent users from querying any MySQL data, change Allow SQL Query to No.

4. (Optional) To prevent users from previewing any MySQL data, change Allow Data Preview to No.

5. At the bottom of the screen, click Next to proceed.

Configure crawler

Before running the MySQL crawler, you can further configure it. (Some of the options may only be available when using the Direct extraction method.)

You can override the defaults for any of these options:

• To select the assets you want to include in crawling, click Include Metadata. (This defaults to all assets, if none are specified.)
• To select the assets you want to exclude from crawling, click Exclude Metadata. (This defaults to no assets if none are specified.)
• To have the crawler ignore tables and views based on a naming convention, specify a regular expression in the Exclude regex for tables & views field.
• For Advanced Config, keep Default for the default configuration or click Custom to configure the crawler:
  • For Enable Source Level Filtering, click True to enable schema-level filtering at source or click False to disable it.
  • For Use JDBC Internal Methods, click True to enable JDBC internal methods for data extraction or click False to disable it.

Did you know?

If an asset appears in both the include and exclude filters, the exclude filter takes precedence.

Run crawler

To run the MySQL crawler:

1. (Direct mode only) To check for any permissions or other configuration issues before running the crawler, click Preflight checks.
2. You can either:
   • To run the crawler once immediately, at the bottom of the screen, click the Run button.
   • To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button.

Once the crawler completes, you can see the assets in Atlan's asset page! 🎉

See also

• How Atlan connects to MySQL: Connection protocols, ports, and security
• What does Atlan crawl from MySQL: Assets and metadata discovered
• Preflight checks for MySQL: Prerequisites and validation