Troubleshooting Databricks errors
When creating or querying foreign Iceberg tables in Databricks Unity Catalog, you may encounter errors related to S3 access, metadata staleness, or missing workspace features. This page covers the most common errors and how to fix them.
S3 access denied when running notebooks
Access denied: The storage credential cannot read from the specified S3 location
Cause
The create or refresh notebook ran before Atlan granted read access to the S3 bucket. The storage credential can't read the Lakehouse metadata files until Atlan updates their IAM trust policy.
Solution
Wait for Atlan to confirm that read access has been granted, then re-run the notebook. See Set up Unity Catalog access for the full setup sequence.
Foreign tables not reflecting latest data
Query results are missing recent assets or show outdated metadata
Cause
The refresh notebook hasn't been run since the Lakehouse data was last updated. Overnight compaction jobs can also delete old Iceberg manifest files that the foreign tables were still pointing to, causing queries to return stale or incomplete results.
Solution
Re-run the refresh notebook to update the metadata pointers:
- Import
dbx_foreign_iceberg_tables_refresh.pyas a Databricks notebook and run it.
To prevent this from recurring, schedule the refresh notebook to run at ~4:00 AM daily, after overnight compactions complete at ~3:00 AM.
Private Preview feature not available on workspace
Unsupported feature: CREATE TABLE ... UNIFORM ICEBERG ... METADATA_PATH is not available
On newer Databricks runtimes, the same underlying limitation can surface as a different error:
CONFIG_NOT_AVAILABLE: spark.databricks.delta.uniform.readIcebergEnabled
Cause
The Foreign Iceberg Tables or Read Iceberg Via Metadata Location Private Preview feature hasn't been enabled on your Databricks workspace. This feature is required before any foreign Iceberg tables can be created.
On newer Databricks runtimes, the spark.databricks.delta.uniform.readIcebergEnabled Spark configuration that the notebook sets is no longer recognized, so the failure appears as CONFIG_NOT_AVAILABLE instead of the Unsupported feature error.
Solution
Contact your Databricks account representative and request enablement of the Foreign Iceberg Tables or Read Iceberg Via Metadata Location Private Preview feature. Databricks uses both names depending on your engine version—refer to whichever your representative recognizes. Once enabled, notify Atlan Support before proceeding. See Enable Databricks private preview.
See also
- Connect Databricks to Lakehouse
- Lakehouse overview: Overview of the Atlan Lakehouse and Iceberg table management.
Need help
If you need assistance after trying these steps, contact Atlan Support.