Oracle assets package

The Oracle assets package crawls Oracle assets and publishes them to Atlan for discovery.

Direct extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl assets directly from Oracle:

Java

Coming soon

Python

Direct extraction from Oracle

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import OracleCrawler

client = AtlanClient()

crawler = (
    OracleCrawler( # (1)
        connection_name="production", # (2)
        admin_roles=[client.role_cache.get_id_for_name("$admin")], # (3)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (4)
        allow_query=True, # (5)
        allow_query_preview=True, # (6)
    )
    .direct(hostname="test.oracle.com", port=1521) # (7)
    .basic_auth( # (8)
        username="test-username",
        password="test-password",
        sid="test-sid",
        database_name="test-db",
    )
    .include(assets={"ANALYTICS": ["SALES", "CUSTOMER"]})  # (9)
    .exclude(assets={})  # (10)
    .exclude_regex("TEST*")  # (11)
    .jdbc_internal_methods(True)  # (12)
    .source_level_filtering(False)  # (13)
    .to_workflow() # (14)
)
response = client.workflow.run(crawler) # (15)

1. Base configuration for a new Oracle crawler.

2. You must provide a name for the connection that the Oracle assets will exist within.

3. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

4. You can specify a maximum number of rows that can be accessed for any asset in the connection.

5. You can specify whether you want to allow queries to this connection. (True, as in this example) or deny all query access to the connection (False).

6. You can specify whether you want to allow data previews on this connection (True, as in this example) or deny all sample data previews to the connection (False).

7. To configure the crawler for extracting data directly from Oracle then you must provide the following information:

   • hostname of your Oracle instance.
   • port number of the Oracle instance (use 1521 for the default).

8. When using basic_auth(), you must provide the following information:

   • username through which to access Oracle
   • password through which to access Oracle
   • SID (system identifier) of the Oracle instance
   • database name to crawl

9. You can also optionally specify the set of assets to include in crawling. For Oracle assets, this should be specified as a dict keyed by database name with values as a list of schemas within that database to crawl. (If set to None, all databases and schemas will be crawled.)

10. You can also optionally specify the list of assets to exclude from crawling. For Oracle assets, this should be specified as a dict keyed by database name with values as a list of schemas within the database to exclude. (If set to None, no assets will be excluded.)

11. You can also optionally specify the exclude regex for crawler ignore tables and views based on a naming convention.

12. You can also optionally specify whether to enable (True) or disable (False) JDBC internal methods for data extraction.

13. You can also optionally specify whether to enable (True) or disable (False) schema level filtering on source, schemas selected in the include filter will be fetched.

14. Now, you can convert the package into a Workflow object.

15. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Offline extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

To crawl Oracle assets from the S3 bucket:

Java

Coming soon

Python

Crawl assets from the S3 bucket

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import OracleCrawler

client = AtlanClient()

crawler = (
    OracleCrawler( # (1)
        client=client, # (2)
        connection_name="production", # (3)
        admin_roles=[client.role_cache.get_id_for_name("$admin")], # (4)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (5)
        allow_query=True, # (6)
        allow_query_preview=True, # (7)
    )
    .s3( # (8)
        bucket_name="test-bucket",
        bucket_prefix="test-prefix",
    )
    .jdbc_internal_methods(True)  # (9)
    .source_level_filtering(False)  # (10)
    .to_workflow() # (11)
)
response = client.workflow.run(crawler) # (12)

1. Base configuration for a new Oracle crawler.

2. You must provide a client instance.

3. You must provide a name for the connection that the Oracle assets will exist within.

4. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

5. You can specify a maximum number of rows that can be accessed for any asset in the connection.

6. You can specify whether you want to allow queries to this connection. (True, as in this example) or deny all query access to the connection (False).

7. You can specify whether you want to allow data previews on this connection (True, as in this example) or deny all sample data previews to the connection (False).

8. When using s3(), you need to provide the following information:

   • name of the bucket/storage that contains the extracted metadata files.
   • prefix is everything after the bucket/storage name, including the path.

9. You can also optionally specify whether to enable (True) or disable (False) JDBC internal methods for data extraction.

10. You can also optionally specify whether to enable (True) or disable (False) schema level filtering on source, schemas selected in the include filter will be fetched.

11. Now, you can convert the package into a Workflow object.

12. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Secure agent extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

In this example, we demonstrates creating an Oracle Secure Agent workflow with basic authentication (OracleCrawler.AuthType.BASIC). You can create a similar workflow for Kerberos authentication (OracleCrawler.AuthType.KERBEROS) by providing the necessary configuration.

Java

Coming soon

Python

To crawl Oracle assets in the offline agent

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.packages import OracleCrawler

client = AtlanClient()

crawler = (
    OracleCrawler( # (1)
        client=client, # (2)
        connection_name="production", # (3)
        admin_roles=[client.role_cache.get_id_for_name("$admin")], # (4)
        admin_groups=None,
        admin_users=None,
        row_limit=10000, # (5)
        allow_query=True, # (6)
        allow_query_preview=True, # (7)
    )
    .agent_config( # (8)
        hostname="test.oracle.com",
        port=1521,
        auth_type=OracleCrawler.AuthType.BASIC,
        default_db_name="test-db",
        sid="test-sid",
        agent_name="test-agent",
        aws_region="us-east-1",
        aws_auth_method=OracleCrawler.AwsAuthMethod.IAM,
        secret_store=OracleCrawler.SecretStore.AWS_SECRET_MANAGER,
        secret_path="test/path/secret.json",
        agent_custom_config={"test": "config"},
    )
    .include(assets={"ANALYTICS": ["SALES", "CUSTOMER"]})  # (9)
    .exclude(assets={})  # (10)
    .jdbc_internal_methods(True)  # (11)
    .source_level_filtering(False)  # (12)
    .to_workflow() # (13)
)
response = client.workflow.run(crawler) # (14)

1. Base configuration for a new Oracle crawler.

2. You must provide a client instance.

3. You must provide a name for the connection that the Oracle assets will exist within.

4. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users).
   • a list of groups (names) that will be connection admins.
   • a list of users (names) that will be connection admins.

5. You can specify a maximum number of rows that can be accessed for any asset in the connection.

6. You can specify whether you want to allow queries to this connection. (True, as in this example) or deny all query access to the connection (False).

7. You can specify whether you want to allow data previews on this connection (True, as in this example) or deny all sample data previews to the connection (False).

8. When using secure agent_config(), you need to provide the following information:

   • host address of the Oracle instance.
   • port number for the Oracle instance. Defaults to 1521.
   • authentication type (basic or kerberos). Defaults to basic.
   • default database name.
   • SID (system identifier) of the Oracle instance.
   • name of the agent.
   • AWS region where secrets are stored. Defaults to us-east-1.
   • AWS authentication method (iam, iam-assume-role, access-key). Defaults to iam.
   • secret store to use (e.g AWS, Azure, GCP, etc)
   • (optional) path to the secret in the secret manager.
   • (optional) Custom JSON configuration for the agent.

9. You can also optionally specify the set of assets to include in crawling. For Oracle assets, this should be specified as a dict keyed by database name with values as a list of schemas within that database to crawl. (If set to None, all databases and schemas will be crawled.)

10. You can also optionally specify the list of assets to exclude from crawling. For Oracle assets, this should be specified as a dict keyed by database name with values as a list of schemas within the database to exclude. (If set to None, no assets will be excluded.)

11. You can also optionally specify whether to enable (True) or disable (False) JDBC internal methods for data extraction.

12. You can also optionally specify whether to enable (True) or disable (False) schema level filtering on source, schemas selected in the include filter will be fetched.

13. Now, you can convert the package into a Workflow object.

14. Run the workflow by invoking the run() method on the workflow client, passing the created object.

    Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Re-run existing workflow

To re-run an existing workflow for Oracle assets:

Java

Coming soon

Python

Re-run existing Oracle workflow

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.enums import WorkflowPackage

client = AtlanClient()

existing = client.workflow.find_by_type(  # (1)
  prefix=WorkflowPackage.ORACLE, max_results=5
)

# Determine which Oracle workflow (n)
# from the list of results you want to re-run.
response = client.workflow.rerun(existing[n]) # (2)

1. You can find workflows by their type using the workflow client find_by_type() method and providing the prefix for one of the packages. In this example, we do so for the TableauCrawler. (You can also specify the maximum number of resulting workflows you want to retrieve as results.)

2. Once you've found the workflow you want to re-run, you can simply call the workflow client rerun() method.

   • Optionally, you can use rerun(idempotent=True) to avoid re-running a workflow that is already in running or in a pending state. This will return details of the already running workflow if found, and by default, it is set to False.
   Workflows run asynchronously

Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed. :::

Kotlin

Coming soon

Raw REST API

Requires multiple steps through the raw REST API

1. Find the existing workflow.
2. Send through the resulting re-run request.

POST /api/service/workflows/indexsearch

{
  "from": 0,
  "size": 5,
  "query": {
    "bool": {
      "filter": [
        {
          "nested": {
            "path": "metadata",
            "query": {
              "prefix": {
                "metadata.name.keyword": {
                  "value": "atlan-oracle" // (1)
                }
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "metadata.creationTimestamp": {
        "nested": {
          "path": "metadata"
        },
        "order": "desc"
      }
    }
  ],
  "track_total_hits": true
}

1. Searching by the atlan-oracle prefix will ensure you only find existing Oracle assets workflows.

   Name of the workflow

The name of the workflow will be nested within the _source.metadata.name property of the response object. (Remember since this is a search, there could be multiple results, so you may want to use the other details in each result to determine which workflow you really want.) :::

POST /api/service/workflows/submit

{
  "namespace": "default",
  "resourceKind": "WorkflowTemplate",
  "resourceName": "atlan-oracle-1684500411" // (1)
}

1. Send the name of the workflow as the resourceName to rerun it.