Skip to main content

What does Atlan crawl from Coalesce Private Preview

This page is the complete inventory of what the Atlan Coalesce connector ingests, how it surfaces in the catalog, and where the boundaries of the current capability are.

Hierarchy ingested

Atlan reads the following hierarchy from Coalesce on every crawl:

  1. Projects: every project the API token can access (or only those matched by the include/exclude filter).
  2. Environments: every environment within each project.
  3. Nodes: every node within each environment (Source, Stage, View, and other Coalesce node types).
  4. Runs: the most recent Refresh runs in each environment, walked from newest to oldest within the lookback window (7 days by default).
  5. Per-node run results: the run state of every node included in each walked run.

Attributes written on Snowflake assets

Coalesce metadata is written directly onto the underlying Snowflake table or view that Coalesce materializes. The connector doesn't create separate Coalesce-typed entities in the catalog.

For every Snowflake asset that Coalesce manages, the following eight attributes are populated:

Atlan fieldSource fieldData typeDescription
sqlCoalesceLastRunStatusCoalesce runState, normalizedStringCoalesce's run state mapped to a standard Atlan status: one of success, failure, skipped, canceled, unknown.
sqlCoalesceNodeStatusCoalesce runState (raw)StringRaw Coalesce run state for the node (for example complete, error, skipped).
sqlCoalesceLastRunAtCoalesce runEndTime (per node)Long (epoch ms)Timestamp of the run that produced this node's status. Each node carries its own timestamp, since different nodes may come from different runs in multi-job environments.
sqlCoalesceNodeTypeCoalesce nodeTypeStringCoalesce node type, for example Source, Stage, View.
sqlCoalesceEnvironmentIdCoalesce environment IDStringIdentifier of the Coalesce environment containing this node.
sqlCoalesceEnvironmentNameCoalesce environment nameStringDisplay name of the Coalesce environment.
sqlCoalesceProjectIdCoalesce project IDStringIdentifier of the Coalesce project containing the environment.
sqlCoalesceProjectNameCoalesce project nameStringDisplay name of the Coalesce project.

Lineage

Atlan emits Process and ColumnProcess entities representing the Coalesce transformation between each node and its Snowflake target.

  • Process: one per non-Source Coalesce node. Inputs are the upstream Coalesce nodes (resolved via Coalesce's sourceMapping). Outputs are the Snowflake table or view materialized by the node.
  • ColumnProcess: column-level lineage for the same transformation, where Coalesce provides column-level sources.

Announcements

When the most recent Coalesce run for a node has a non-success state (error, skipped, canceled), Atlan writes an announcement banner on the corresponding Snowflake table or view. The banner includes:

  • The Coalesce run ID
  • The Coalesce run type (Refresh, etc.)
  • The node name and environment name
  • The error message from the run results, if any

If a node was successfully executed in Coalesce as part of its latest run (run state complete), any existing announcement banner for that node is cleared.

Lookback window: Last 7 days

Per-node status reflects the most recent Refresh run that touched each node within the last 7 days.

  • Nodes that have been run within the window → status, timestamp, and announcement (if applicable) are populated.
  • Nodes that haven't been run in the last 7 days → status fields appear blank in the UI.
  • Nodes that have never been run → status fields appear blank.

The window is configurable per connector instance via run_status_lookback_days. Customers with monthly-cadence Coalesce jobs can request an extended window. Contact your Atlan team.

Status assignment across multiple jobs

A Coalesce environment commonly hosts multiple jobs: a Deploy job, plus one or more Refresh jobs scoped to different domains or schedules. A given Snowflake table may be touched by more than one job. In this case:

  • Atlan records each node's status from the most recent run that included it, across all jobs in the environment.
  • If Job A ran most recently and touched the node, the node's status comes from Job A's latest run.
  • If a different node was last touched by Job B (which ran earlier), that node's status comes from Job B's latest run.

Each node shows its most recent run result.

Current crawl limitations

The following are out of scope for the Coalesce connector today:

ItemReason
Deploy run status (DDL or schema-change runs)The connector intentionally pulls Refresh run state only, since the latest Refresh run is the signal customers care about for data freshness. Surfacing Deploy run status requires an additional set of API calls per environment.
In-progress runsAtlan pulls run state from the Coalesce API. A completed run carries a definitive final status, while an in-progress run doesn't, so pulling completed status is more reliable than reading an in-progress status. The connector skips in-progress runs and picks them up on the next crawl after they complete.
Real-time push updatesCoalesce's public API is request-response, so Atlan refreshes on a schedule rather than receiving live push updates.
Coalesce-only assetsThe connector enriches existing Snowflake assets only as of today. Coalesce nodes that don't materialize into Snowflake aren't surfaced in Atlan.
Coalesce user/role metadataThe connector reads structural and run metadata only. Coalesce account/user data isn't ingested.
Custom Coalesce attributesAtlan writes a fixed set of eight sqlCoalesce* attributes. Customer-defined Coalesce metadata fields aren't surfaced.

Practical scale

To return per-node status, the connector makes multiple Coalesce API calls per environment: one to list recent runs, then one per walked run to fetch per-node results. For larger workspaces this means many sequential API calls, which can extend total crawl time. The connector has been validated end-to-end on workspaces with ~2,000 nodes.

If your workspace is significantly larger or you are seeing slow crawls, reach out to your Atlan customer success manager or open a support ticket for guidance on crawl cadence and rate-limit coordination. See also Set up Coalesce: rate limits and large workspaces.

Where Coalesce metadata appears in Atlan

Coalesce metadata enriches the underlying Snowflake assets it transforms—it doesn't create separate Coalesce assets in the catalog. On any Snowflake table or view materialized by Coalesce, you can see:

  • The Coalesce Node Status panel on the overview tab (run state, last run timestamp, environment, project).
  • Eight sqlCoalesce* attributes on the asset (last run status, node status, last run timestamp, node type, environment ID and name, project ID and name).
  • An announcement banner if the most recent Coalesce run for that node failed, was skipped, or was canceled.
  • Lineage edges between Coalesce-managed nodes and their Snowflake materializations.

For this enrichment to appear, your Snowflake connector must already be installed and have crawled successfully.

See also

  • Set up Coalesce: Configure authentication and permissions before running the first crawl.