ZenotiHelp

Data freshness and refresh cycles

ZTZenoti Docs Team Updated April 10, 2026 4 min readGuide

Analytics β€” Amazon S3 delivers data through a cycle of full loads and incremental updates. Understanding how data refreshes work helps you design reliable pipelines and avoid querying incomplete data.

Before you begin

  • Your organization must have Analytics β€” Amazon S3 enabled and configured.
  • The initial full load must be complete before incremental refresh cycles begin.

Full load vs incremental load

There are two types of data loads:

Load type GroupID When it runs What it contains
Full load 1 Once, during initial onboarding All historical data from your Zenoti organization
Incremental load 2, 3, 4,... Every 1–2 hours after onboarding Only records created, updated, or deleted since the previous cycle

The full load gives you a complete historical baseline. Every incremental load adds a new GroupID folder containing only the changes from that cycle.

What is GroupID

GroupID is a serial number that identifies each data refresh cycle. Every time the system runs a data refresh, it creates a new GroupID folder in your S3 bucket under the PODName folder.

  • GroupID 1 β€” always the initial full load. Contains all historical data.
  • GroupID 2, 3, 4,... β€” each subsequent incremental load. The higher the number, the more recent the data.

When building a downstream pipeline, process GroupID folders in ascending order. The smallest GroupID contains your oldest data; the largest contains your most recent changes.

Data refresh cadence

Incremental data refreshes run approximately every 1–2 hours. Actual timing varies depending on the volume of data being processed and overall system conditions. Do not rely on an exact interval.

Each refresh cycle:

  1. Identifies records that changed in the Zenoti operational system since the last cycle.
  2. Processes those changes through the data pipeline.
  3. Delivers the results as a new GroupID folder in your S3 bucket.

The processing marker

While a data refresh is in progress, the GroupID folder appears with a _processing suffix:

PODName/
  4_processing/     <- load is still running
  3/                <- completed, safe to query
  2/                <- completed, safe to query
  1/                <- completed, safe to query

When the refresh completes successfully, the folder is renamed to remove the suffix:

PODName/
  4/                <- now complete, safe to query
  3/
  2/
  1/

Important: Never read from a folder that ends in _processing. The data is incomplete. Always wait for the suffix to be removed before querying that GroupID.

Detecting new data in your pipeline

Every table includes ETL metadata columns that you can use to identify records processed in a specific refresh cycle:

Column Description
etlcreateddate Timestamp when this record was first written by the data refresh
EtlLastUpdatedOn Timestamp when this record was last updated by the data refresh
etlcreatedby Identifier for the data refresh process that created the record

To process only records added since your last sync, use etlcreateddate as a watermark:

SELECT *
FROM Bi_FactInvoiceItem
WHERE etlcreateddate > @LastSyncTimestamp
  AND void = 0;

Replace @LastSyncTimestamp with the etlcreateddate value from the last record you processed in your previous run.

How fact table updates work β€” delete-insert mechanism

Fact tables use a delete-insert mechanism to handle updates and deletions from the source system:

  • void = 0 β€” the record is current and active. Include it in your queries.
  • void = 1 β€” the record was deleted or updated in the source system. Remove it from your downstream data.

When you consume a new incremental GroupID:

  1. Find any records with void = 1 in the new GroupID.
  2. Remove those records from your downstream store (match on the natural key or the source record identifier).
  3. Insert all records with void = 0 from the new GroupID.

Always filter WHERE void = 0 when reading from fact tables for reporting. If you do not filter, your results will include stale and deleted records.

How dimension table updates work β€” update-insert mechanism

Dimension tables use a different pattern:

  • Updated records replace existing rows in the dimension table.
  • New records are inserted as new rows.

You do not need to manage a void column for dimension tables. The latest version of each record is always present.

Monthly snapshots

In addition to the regular incremental cycle, Analytics β€” Amazon S3 generates monthly snapshots of your fact data. Snapshots capture the state of your data as of the 1st of each month.

Snapshot folder structure:

CustomerBucket/
  MonthlySnapshot/
    ExtractDate/
      PodName/
        GroupID/
          FactName/
            logid.zip

Monthly snapshot GroupIDs start in the 1000 series (for example, 1001, 1002, 1003), so you can distinguish them from regular incremental GroupIDs at a glance.

Notes about monthly snapshots:

  • Snapshots are available only after all centers in your organization have been onboarded and historical data has been fully loaded.
  • The following tables are excluded from monthly snapshots: Bi_FactEmployeeMembershipCommissions, Bi_FactItemCommissionDeductions, Bi_FactMultipleEmployeeCommission, Bi_FactAdoptionSummary, Bi_FactOpportunityCustomFieldValues, Bi_DimUtcToCenterTime, Bi_FactCampaignSummary, Bi_FactEmployeeItemLevelCommissions.
Was this article helpful?
Your feedback helps us improve our documentation.