S3 folder structure and file format
Navigate and understand the hierarchical folder structure and CSV file format used in your Amazon S3 bucket for Analytics β Amazon S3 data delivery.
Before you begin
- Your organization must have Analytics β Amazon S3 enabled and configured.
- You need access credentials to your organization's S3 bucket.
- At least one full data load must be completed before folders and files appear in your bucket.
Steps
Navigate to your S3 bucket root directory
- Connect to your S3 bucket using your provided access credentials.
- Navigate to your organization's root directory folder.
- Look for one or more PODName folders within the root directory.
Note: If your business operates across multiple geographies, you may see multiple PODName folders. Each PODName represents a different geographic region or data pod.
Understand the folder hierarchy
The S3 bucket follows this hierarchical structure:
Root Directory/
PODName/
GroupID/
table_name/
logid/
filename.csv
Each level serves a specific purpose:
- Root Directory β Your organization's topmost folder.
- PODName β Geographic region or data pod identifier.
- GroupID β Sequential number indicating the data load cycle.
- table_name β Individual table folder (e.g.,
Bi_FactInvoiceItem). - logid β Sequential processing identifier for the data load.
- filename.csv β The actual CSV data file.
Navigate GroupID folders for data loads
- Within your PODName folder, look for numbered GroupID folders.
- Start with GroupID 1 for your initial full data load.
- Access higher-numbered GroupIDs (2, 3, 4, etc.) for incremental data updates.
- Avoid any folders named with a
_processingsuffix β these indicate data loads in progress.
Important: Only query completed GroupID folders. Folders with
_processingsuffixes are temporary and are renamed when the data load completes.
Access table data within GroupID folders
- Navigate into a GroupID folder to see all available table folders.
- Open the specific table folder you need (e.g.,
Bi_DimUserfor guest data). - Navigate into the logid subfolder within the table folder.
- Download the CSV files from the logid folder.
Note: In typical operation, there is one logid folder per table per GroupID. Multiple logid folders may exist after manual re-runs or data corrections.
Work with CSV files
- Download CSV files. File names follow the format
<logidnumber-fivedigitserialnumber>.csv. - For large data volumes, look for multiple files named
Data0.csv,Data1.csv,Data2.csv, etc. - Process files knowing they are UTF-8 encoded and comma-delimited.
- Reference the SchemaFull folder in your bucket root for column definitions β CSV files do not contain headers.
Important: CSV files do not include column headers. The column order is defined by the schema files in the SchemaFull folder at your bucket root.
Tips
- Start with the smallest GroupID β GroupID 1 contains your complete historical data set. Higher GroupIDs contain only changes since the previous load.
- Check multiple PODName folders β If your business spans multiple regions, ensure you access data from all relevant PODName folders.
- Monitor for new GroupIDs β New GroupID folders appear every 1β2 hours as incremental data refreshes complete.
- Use schema files for reference β The SchemaFull folder contains
.txtfiles defining the structure and column order for each table.