Late Data Handeling

Overview

Late data is data that arrives at the Historian with a timestamp earlier than the most recently stored point — out of chronological order. This happens regularly in industrial deployments: a field device has a clock that drifts, a Collector goes offline and buffers data locally, or a store-and-forward replay delivers a backlog of measurements after a network outage.

Timebase accepts and stores late data automatically. The Late Data Tolerance setting gives you control over how far back the Historian will accept late data, and lets you discard small clock-skew duplicates without losing genuine backfill data.

How late data works

Storage and the .late file

When a data point arrives with a timestamp that falls inside a completed hourly block, the Historian cannot write it directly into that block — the block may already be in use by a reader. Instead, the Historian buffers the late point in a temporary .late file alongside the corresponding .data block. During the next maintenance cycle (approximately every minute), the Historian merges all pending .late files back into their .data blocks and deletes the .late files. After merging, the data is available for queries.

You can monitor late data arrival using the Dataset.Writes.Late system tag, which reports the rate of late data ingestion in points per second.

Late Data Tolerance

The Late Data Tolerance setting defines a clock-skew window in milliseconds. Its purpose is to distinguish genuine late data (a backfill) from clock-skew noise (a device whose clock is slightly off and repeatedly sends timestamps a few milliseconds in the past).

Example: With Late Data Tolerance set to 100 ms, a point arriving 50 ms before the most recent stored timestamp is discarded as clock noise. A point arriving 5 seconds before the most recent stored timestamp is accepted as a genuine backfill point.

How to configure

In the Historian browser (Dataset Editor)

1. Navigate to the Historian at http://<historian>:4511.
2. Open the Datasets view.
3. Click the edit icon for the dataset you want to configure.
4. Find the Late Data (ms) field and enter the tolerance in milliseconds.
5. Save the changes.

The Datasets grid also displays the current Late Data Tolerance for each dataset in the Late Data (ms) column.

Via the REST API

Set ldt (Late Data Tolerance in milliseconds) when creating or updating a dataset:

POST /api/datasets/{datasetId}
Content-Type: application/json

{
  "ldt": 100
}

The same field is returned in GET /api/datasets and GET /api/datasets/{datasetId} responses.

Choosing a value

Troubleshooting