Skip to main content

Crawl Talend assets

Create a workflow in Atlan to discover and catalog Talend assets from project files stored in a GitHub or Atlassian Stash repository. The workflow extracts metadata from your Talend project files and creates assets in Atlan for projects, jobs, components, and fields.

Prerequisites​

Before you begin, verify you have:

  • Completed the Set up Talend guide
  • Repository URL and personal access token ready
  • Branch name containing your Talend project files

Create crawler workflow​

Create a new workflow in Atlan to connect to your Talend project repository.

  1. Log in to Atlan.
  2. In the top right of any screen, navigate to New and then click New Workflow.
  3. In the Marketplace tab, search for or locate Talend Connector.
  4. Select Talend Connector to open the workflow setup.
  5. Select GitHub Cloud Store as the cloud store type.

Configure connection​

Configure repository access credentials and connection details.

  1. Enter your repository details in the connection form:
    • Repository URL: Full URL to your Talend project repository (for example, https://github.com/organization/repository)
    • Git access token: Personal access token with read permissions
    • Branch name: Branch to crawl (for example, main or master)
    • Talend project path (optional): Subdirectory path if your Talend projects are in a subfolder (for example, talend_projects/)
  2. Enter a Connection name that represents your Talend environment (for example, production-talend). This name appears in Atlan to identify this Talend connection.
  3. To change who can manage this connection, modify the users or groups listed under Connection Admins.

Run crawler​

Run the workflow to extract metadata from your Talend project files and create assets in Atlan.

  1. After preflight checks pass (or to skip them), choose how to run the workflow:
    • Run: Execute the crawler once immediately
    • Schedule & Run: Schedule the crawler to run on a recurring schedule (hourly, daily, weekly, or monthly)
  2. Select your preferred option to start the workflow. The workflow begins processing your Talend project files. You can monitor progress in the workflow execution details.
  3. After the workflow completes, Talend assets appear in Atlan.

Need help​

If you encounter issues while crawling Talend assets, see Connection and authentication issues for solutions to common workflow failures. For additional assistance, contact Atlan support by submitting a request.

See also​