Migration field mapping and troubleshooting
This reference covers the detailed mapping and troubleshooting work you'll do during a migration. For the framing, sequence, and pitfalls, see the migration best-practice guide.
Custom field mapping
Legacy tools accumulate custom fields. Each one maps to exactly one of three destinations in Atlan—go field by field with your technical lead:
- A native field: if a direct equivalent exists (description, owner, certificate). Preferred: it's searchable and can drive workflows.
- A custom-metadata attribute: structured, typed values (use controlled-vocabulary lists for enums, text for free-form). Keep owner/steward fields distinct if they drive automation. Group custom metadata by source system or domain—don't create a separate set per asset type.
- A README: rich, freeform content, and anything destined for AI or agent consumption.
A few field types have clear defaults: reporting counters and KPI tallies (for example, "number of columns with a term") must be dropped and re-derived dynamically post-migration—they go stale the instant a copy is made. Source-system stable IDs must always be carried as custom metadata—they're your cross-reference key during validation. Build these with the UI for a handful of fields, or with a bulk import for large lists; for 50+ fields, have your team do an offline keep/drop/maybe pass guided by each field's fill-rate before committing.
Content-type mapping
The following are the mapping decisions that consistently trip up migration teams. Work through them before writing any ingestion script.
- Review the extracts column by column before mapping. Legacy CSV/Excel exports contain many catalog-internal columns that must never map anywhere. Walk each extract file live with the people who know the legacy tool, column by column. Before any mapping, request the legacy tool's relationship/operating-model export and production exports per priority asset type—and confirm you're working from production, not a non-prod environment full of test relationships.
- Ownership roles: map by how each role is consumed, not by name. Downstream automation breaks when roles are conflated. Ask how each role field drives automation before choosing its destination: the accountable owner maps to the native owner field; a steward who logs in and edits maps to a custom-metadata user-type attribute; a custodian, team alias, or channel maps to a custom-metadata text or enum field. Don't collapse all roles into the native owner field—ticket routing and automation depend on the distinction.
- Multiple legacy description types: profile before deciding. Run a comparison across all description fields first. If more than ~80% are identical, consolidate into the primary description and discard the redundant copies; if they genuinely serve different audiences, send the business description to the native field and the technical one to custom metadata. Apply an idempotency rule on load—if a description is already human-set in Atlan, never overwrite; populate blanks only. Strip embedded HTML before ingestion.
- Inter-term links: resolve in three buckets. Categorize unresolved legacy term links as (1) exact match—resolve automatically; (2) prefix/partial match—resolve conservatively into description text only and surface for review; (3) dead link—do not create orphan references; surface the list for a keep/drop decision. Never autoapply a fuzzy match to lineage.
- Column-to-column business relationships have no direct equivalent. Atlan links columns through lineage (data flow), not free-form semantic relations. If the legacy relationship is business-conceptual, model the shared concept as a glossary term and link both columns to it. Don't force conceptual relationships into the lineage graph—lineage implies data movement.
- File and flat-file assets need schema-bearing types. If the File asset type in your tenant doesn't carry the per-field schema or column detail you need, model each logical file as a Table and each field as a Column under a generic connection—confirm current File-asset capabilities in the Atlan product documentation. For file-mediated lineage (system A → file → system B), keep the file as a first-class node and build two hops—don't collapse it to A → B, or you lose the audit point.
Object-level mapping
There is no one-to-one mapping between a legacy catalog's object model and Atlan's—build a mapping workbook. As a starting point:
| Legacy concept | Atlan concept |
|---|---|
| Communities / top-level groupings | Domains |
| Curated assets you package for consumers | Data products |
| Relations between assets | Lineage graph |
| Workflows | Playbooks / governance workflows |
| Custom attributes | Custom metadata or tags |
| Business terms and glossaries | Glossaries and terms |
| Policy documentation artifacts | Glossary structure (the policy becomes a glossary, its standards become terms): not Atlan access or metadata policies |
A shared asset can belong to more than one domain, so don't distort your domain model to force a single-home mapping. Where a legacy structure has no clean equivalent, encode the relationship in a naming convention rather than restructuring the taxonomy to fit—and flag the gap rather than over-engineering around it.
Troubleshooting
| Symptom | Likely cause | Next move |
|---|---|---|
| Assets are missing after crawl | Several distinct causes—bucket them before remediating | Get asset-level detail (db/schema/table names), not aggregate counts. Common buckets: expected non-crawl objects (some source objects such as certain partition sub-tables may be excluded from crawl by design—confirm against the connector documentation before writing them off); a naming issue causing a workflow to skip a database (fix the workflow); a crawler failure (review the workflow error log); a workflow that exists but was never scheduled (set recurrence); or a connector scope mismatch where Atlan is configured for fewer schemas than the legacy catalog indexed (compare the include/exclude filter against the extract). Migrate enrichment only for assets that exist; flag the rest as blocked pending scope expansion. Don't remediate an unstable list—wait until it's final. |
| A value shows in the Atlan UI but is absent from an export file | Export artifact, not data loss | Confirm the value is present in the UI, fix the affected records manually, and don't halt validation or re-run migration scripts. |
| Lineage extraction is slow or fails only on large databases | A source-side extraction problem, not an Atlan issue | Don't wait for 100% extraction. If the approach is validated on small databases, request the partial data already extracted and build the lineage import in parallel; scale once the large-DB extraction is fixed. |
| Migrated lineage has drifted | You migrated static lineage instead of regenerating it | Let connectors and miners regenerate lineage dynamically. Static lineage diverges the moment a pipeline changes—treat any static copy as a temporary risk item. |
| A deprecated database lost its lineage arrows | Archived assets show no lineage by design | If the database has no active downstream use, archiving is fine. If it still feeds active downstream tables, load it as an active connection with a name that signals deprecation, so lineage arrows are preserved. Confirm which case each database is before deciding. |
| A self-built (your own) migration script fails | Most often an API key without the required permissions; also misconfigured host URL, or a language-version incompatibility | Don't diagnose from a verbal description. Reproduce the exact script in a controlled sandbox and compare line by line; check the key's permissions and the host-URL format first. |
| A bulk import fails with no visible error | A silent validation failure | Test small batches first, and pull the underlying workflow logs. When escalating, provide tenant name, asset type and count, repro steps, error messages, screenshots, and a workflow ID or timestamp. |
| The connection-to-connection migration utility runs but changes nothing | The utility matches assets by qualified name, so if names differ between source and target it may not match—see the migration-tooling documentation for its current matching behavior | When names differ, switch to a one-time asset export/import. Build an old-to-new qualified-name mapping (fuzzy lookup plus AI-assisted description matching for the stragglers), update qualified names in the export file, and import. |
| A source's auth pattern (for example an external IAM role or a non-default port) isn't handled by the connector | Possibly a connector coverage gap rather than a configuration error—confirm current connector auth support in the product documentation | Check whether an analogous connector already supports the pattern; raise a formal feature request with the specific pattern; and scope around it by progressing the cleared connectors—don't let one source block the wave. |
| Enrichment validated in a lower environment needs to move to production | Environments are separate; enrichment doesn't promote automatically | Use a scoped asset export (assets + the attributes you enriched) from the lower environment, map to the production asset identifiers, and re-import. Confirm any referenced glossary terms exist in both environments first. |
For the mechanics and current availability of specific migration utilities, bulk-import formats, and cross-environment packages, see the Atlan product and migration-tooling documentation—those specifics change and aren't settled here.
Related
- Migration best practices: framing, sequence, and pitfalls.
- Connector scoping, crawl filters, and why lineage is missing or partial: connector & ingestion best practices.
- How lineage is generated per source, and impact analysis: lineage best practices.