Plan your connectors and metadata ingestion
Getting metadata into Atlan correctly is a sequencing problem, not a configuration problem. Pick the right ingestion mechanism, scope the crawl before you run it, decide how many connections to create, and always crawl before you mine. This guide covers those decisions and how to unblock the common failures. For per-source auth methods, minimum grants, and the current list of supported connectors, follow the connector setup docs—this page is about how to approach the work.
When to use what
A handful of building blocks do almost all the work. Match the simplest one that satisfies the requirement—don't default to "a connector is needed" or "it's all custom."
| Building block | What it does | Use when |
|---|---|---|
| Crawler | Ingests object metadata (schemas, tables, columns, dashboards, jobs); captures table→view / view→view within a system, and warehouse→BI for supported BI pairs | Always, first. This is your baseline catalog. |
| Miner (online) | Adds table→table lineage and popularity from query/usage logs within a system | After the crawler, when you need intra-system lineage or popularity |
| Offline / generic miner | Parses query history you export to object storage | On-prem, strict network boundaries, or historical backfill—intra-system only |
| Event-driven ingestion (for example, OpenLineage) | Adds lineage from orchestrators and compute engines via an event-emitting library installed in the source—no crawler | When the source pushes run events (orchestrators, compute jobs) rather than exposing a queryable catalog; confirm the supported event-driven sources in the connector setup docs |
| Custom lineage (SDK/API, mapping CSV, or a name-match generator) | Defines edges between assets that already exist in the catalog | Cross-system hops, transformations Atlan can't parse, placeholder source nodes |
Choosing a custom-lineage method: source and target names match or are similar (for example, a like-for-like migration) → use the regex name-match generator; names differ completely → use a mapping-CSV builder; you need full control and your team codes → use the API/SDK. Custom lineage links existing assets only—if an endpoint isn't yet crawled the edge won't resolve, so always run a preview/validation pass and verify both endpoints exist before generating at scale.
No native connector for a source? Evaluate in order:
- Does the source already feed a system Atlan connects to (for example, it lands in your warehouse or lakehouse)? Ingest via the intermediate.
- Does its schema drift often? Use the SDK plus a coding assistant to generate a sync script.
- Is it stable and rarely-changing? Import assets from CSV/YAML and add custom lineage.
Confirm whether a native connector exists in the connector docs before assuming you need a custom path.
Recommended sequence
Crawl before you mine—always. The crawler ingests object metadata; the miner derives table-to-table lineage and popularity from query logs on top of it. Running a miner before the crawler completes produces broken or empty lineage.
Canonical order across a stack (each step means running the corresponding Atlan connector or workflow, not the external tool itself):
- Crawl core data stores (databases, event buses).
- Crawl data-quality tools you want represented.
- Mine query logs.
- Ingest extract/load-tool metadata (for example, Fivetran; for event-driven orchestrators like Airflow or Spark, use the event-driven mechanism described earlier rather than a crawler).
- Ingest transformation-tool metadata (for example, dbt).
- Crawl BI tools last, so warehouse assets exist for lineage to stitch onto.
Separate "connect" from "what to crawl." A connection is a single configured, authenticated instance of a connector—you can create several connections from one connector. Establish auth and network on the connection first, then decide crawl scope. Use include/exclude filters to keep dev, UAT, and test schemas out of the catalog—this controls noise and keeps you under any contractual asset limits. Count assets per schema before a narrowing rescan. Reframe "connect everything" to "connect what serves the first use case," and sequence large, noisy sources after smaller ones are stable so your team learns filtering patterns on lower-risk sources.
Run tracks in parallel at kickoff, not sequentially:
- Connector track: raise the internal tickets your DevOps/security team needs (network model per source: public, private link, or a self-deployed runtime for isolated sources) before the instance is even live. Surface long-lead items early: private link for BI tools needs your IT team, and SSO/SCIM setup must start on day one.
- Enrichment track: seed a glossary from existing docs; frame it as validation, not creation.
- Rollout track: identify the first cohort and the connector combination (typically BI + warehouse) that generates internal advocacy.
Distinguish a technical-user rollout (possible once connectors are live and lineage is visible) from a business-user rollout (needs enrichment in a validatable state). Don't collapse everything into one dependency chain—proceed with rollout for already-crawled connectors while a slow one is still in progress.
Before an AI-enrichment cohort: confirm the minimum context stack is live—core warehouse connector, a BI tool, and a running lineage miner (with popularity where the source supports it). Running enrichment agents on a broken lineage state produces poor output.
Get all assets into production directly. Use a dev/staging instance only for validating configs (SSO, workflows), not as a promote-to-prod pipeline—enriched assets carry instance-specific identifiers and don't natively promote between tenants (glossary terms are the exception that exports cleanly). Enrich in production.
How many connections do you need?
Don't mirror a legacy catalog 1:1—that fragments lineage. Atlan doesn't stitch lineage across separate connections automatically, so splitting one system across many connections means cross-connection hops need custom lineage. Unless a hard compliance requirement forces a split, consolidate by environment (prod / non-prod) or by business domain.
Reason across three axes when the default doesn't apply:
- Lineage completeness: fewer connections preserve cross-project and intra-system lineage.
- Governance complexity: more connections multiply the persona/policy surface.
- Team autonomy: data-mesh teams may self-manage their own crawlers.
For unavoidable cross-connection hops, stitch the critical paths with custom lineage. Don't leave this question open—teams over-engineer connection setup when there's no default.
Common pitfalls
- Running a miner before the crawler completes → broken or empty lineage. Never mark a connector "live" until the miner has run on top of a successful crawl and lineage renders in the UI.
- Implying connectors autobridge cross-system lineage. Native cross-system lineage covers only the pairs documented in the lineage docs—check the current list before scoping (historically warehouse→BI pairs). Hops outside that list (source→source, database→warehouse) need custom lineage—state this before you scope connectors, not after.
- Crawling everything "and filtering later" without knowing the connector's actual filter granularity → catalog noise, blown asset limits, lost trust. If a connector's filters can't exclude what you need, don't rename production objects to fit—use an alternative scope or post-crawl filtering.
- Treating an empty asset view as a crawl failure before ruling out persona/filter masking—the assets may just be hidden from the default view.
- Assuming a "success" run status means lineage resolved: an edge won't resolve if an endpoint isn't crawled, so verify both endpoints exist and edges rendered.
- Carrying elevated dev permissions into production instead of resetting to least privilege at the production gate.
- Mirroring a legacy catalog 1:1 for connections—consolidate by environment or domain instead.
- Re-running a runaway workflow (a miner running for many hours, a log explosion, or a miner scanning all projects when configured to scan one) instead of stopping it and collecting logs—a re-run can orphan assets and corrupt lineage.
- Enriching in a staging tenant expecting to promote to prod: there's no native asset promote across tenants; only glossary terms cross over.
Troubleshooting
Diagnose which side owns the problem before you "fix" anything: a failing connector is either a permission/network gap on your side or a product-side bug, and the fix path is entirely different. Capture the raw error from the browser network tab or reproduce with a manual call using the service credential—if the manual call succeeds but the UI doesn't populate, it's product-side (open a support ticket with the captured headers); if it fails, it's a grant or network gap on your side.
| Symptom | Likely cause | Next move |
|---|---|---|
| Empty asset view after a "successful" crawl | Persona/filter masking, then schema filter or grants | Switch to "all assets" / remove the persona filter first, then check the include-filter and grants |
| Lineage missing or partial | Miner not run, only one endpoint crawled, or a cross-system hop | Confirm the crawler ran, the miner ran on top, both endpoints are crawled, and the path isn't a cross-system hop needing custom lineage |
| Large BI crawl times out on the first run | First run is a full scan | Scope the first run down with project filters, pause non-critical workflows, and expect delta (not full) runs afterward |
| Private-link / network auth failure with little error detail | Both-direction connectivity, FQDN/region, or endpoint config | Confirm connectivity both ways and the endpoint; for a warehouse dropdown that's empty, suspect missing warehouse access rather than credentials |
| BI tool ingested but no lineage | Different connector variant, or a metastore that doesn't expose upstream lineage | Confirm you're running the right connector variant for that platform; some backing metastores limit upstream lineage |
| Choosing between BI crawl modes | Some BI connectors offer more than one crawl mode that trades breadth of coverage against depth of lineage and setup overhead | Fewer workspaces → the richer-lineage mode; many workspaces → the broader-coverage mode. Confirm the modes, their exact lineage tradeoffs, and the required permissions for your BI platform in the connector setup docs |
| A transformation-tool crawl picks up zero assets | Missing prerequisite (artifacts not uploaded, wrong path, env mismatch) | Confirm the required artifacts are uploaded to the expected location |
| Slow crawl (many hours) | Extraction vs publish/miner contention, or first-run overhead | Separate extraction from publish time, stagger concurrent workflows, and wait 2–3 runs before optimizing |
| Secrets errors ("failed to fetch secret," JSON parse) | Path-format or escaping | Fix the secret path/format; for production prefer a secret manager or workload identity over plain-text keys |
The deletion-threshold circuit breaker fails a workflow fast when too high a percentage of previously-ingested assets disappear from the source—it's a safety feature that prevents a mass wipe of enriched assets, not a config error. If a large deletion is genuinely intentional (migration, cleaned-up dev jobs), ask support to raise the deletion threshold on the affected workflow for the duration of the migration, then revert it once the migration completes. If you're narrowing scope and most assets are deleted, it's cleaner to delete the connection and re-run with the correct include-filter than to fight the guardrail. Never delete the only workflow on a connection to "start fresh": that deletes the connection and all its configuration.
When a blocker is on your side (network, firewall, DBA, or security capacity), keep progressing the parallel unblocked workstreams and route the blocker to the owning team with the exact written requirement—the IP to whitelist, the DNS format, the specific grant—and a clear trigger for when work resumes. Avoid open-ended status checks; give the owning team a concrete ask.
For per-source auth methods, minimum grants, IP/egress requirements, and secret-store setup, follow the connector setup docs: don't diagnose those from memory.
Related
- Lineage: capturing, reading, and troubleshooting lineage once connectors are live.
- Data quality: monitoring the assets a connector brings in.
- Connector setup docs and the supported-connector reference: per-source auth, grants, network requirements, and the current connector matrix.