S3 Inventory Report Structure

✦TL;DR

Expected folder structure and format for S3 inventory reports used by Atlan's S3 crawler for inventory-based ingestion.

||

|Connect

This reference outlines the expected folder layout and file format for Amazon S3 inventory reports used by Atlan’s S3 crawler when running inventory-based ingestion.

important

The crawler supports a single destination bucket, with an optional prefix, to store all inventory reports from multiple source buckets.

Folder structure

To enable successful inventory-based crawling, your destination bucket must follow this structure once inventory reports are generated:

📦 <destination-bucket>/
├── [📁 <optional-prefix>/] (applies to all buckets)
│   ├── 📁 <source-bucket-1>/
│   │   ├── 📁 <inventory-config-1>/
│   │   │   ├── 📁 <YYYY-MM-DDTHH-MM-Z>/ (timestamp folders)
│   │   │   │   └── 📄 manifest.json
│   │   │   └── 📁 data/ (CSV or gzipped files)
│   │   │       └── 📄 <inventory-file>.csv.gz
│   │   └── 📁 <inventory-config-2>/
│   └── 📁 <source-bucket-2>/
└── ...

Required	Component	Description	Example
✅	Destination bucket	Single S3 bucket to store all inventory reports	atlan-inventory-reports
❌	Prefix	Folder prefix to organize reports (same for all source buckets)	inventory-reports
✅	Source bucket folder	Folder named after each source bucket	source-bucket-1
✅	Inventory config folder	Folder named after the inventory configuration	daily-inventory
✅	Timestamp folder	Folder with report generation timestamp	2024-01-16T00-00Z
✅	Manifest file	manifest.json containing report metadata	manifest.json
✅	Data folder	Folder containing compressed inventory files	data
✅	Inventory files	.csv.gz or .parquet files with actual data	inventory-2024-01-16-00-00Z.csv.gz

Examples

📁 Basic structure (single source bucket)

📦 atlan-inventory-reports/
└── 📁 source-bucket-1/
    └── 📁 inventory-config-1/
        ├── 📁 2024-01-16T00-00Z/
        │   └── 📄 `manifest.json` {/* Required metadata file */}
        └── 📁 data/
            └── 📄 inventory-2024-01-16-00-00Z.csv.gz

📁 Multiple source buckets

📦 atlan-inventory-reports/
├── 📁 source-bucket-1/
│   └── 📁 inventory-config-1/
│       ├── 📁 2024-01-16T00-00Z/
│       │   └── 📄 `manifest.json` {/* Required metadata file */}
│       └── 📁 data/
│           └── 📄 inventory-2024-01-16-00-00Z.csv.gz
└── 📁 source-bucket-2/
    └── 📁 inventory-config-1/
        ├── 📁 2024-01-16T00-00Z/
        │   └── 📄 `manifest.json` {/* Required metadata file */}
        └── 📁 data/
            └── 📄 inventory-2024-01-16-00-00Z.csv.gz

📁 With optional prefix

📦 atlan-inventory-reports/
└── 📁 inventory-reports/
    ├── 📁 source-bucket-1/
    │   └── 📁 inventory-config-1/
    │       ├── 📁 2024-01-16T00-00Z/
    │       │   └── 📄 `manifest.json` {/* Required metadata file */}
    │       └── 📁 data/
    │           └── 📄 inventory-2024-01-16-00-00Z.csv.gz
    └── 📁 source-bucket-2/
        └── 📁 inventory-config-1/
            ├── 📁 2024-01-16T00-00Z/
            │   └── 📄 `manifest.json` {/* Required metadata file */}
            └── 📁 data/
                └── 📄 inventory-2024-01-16-00-00Z.csv.gz

See also

• Set up inventory reports for S3: Set up inventory reports for S3