Datasheet Historian File Formats

Datasheet Historian File Formats

Tani Historian File Description

Directory structure

The historian has a configurable data directory. All historian variables that use file storage write the data in this directory. It can be changed in <install directory>/Historian/bin/Historian.Settings, key "HistorianFilePath".

Below this directory, each historian variable has its own sub-directory (name is the variable name). Each directory contains these files:

Var.ini contents

The Var.ini file is an ini-File with one section. The section name is "[Var.<variable name>]". In this section, the following keys can exist:

Active
Boolean; yes enables the variable in the runtime system.
AggregateAvg
Boolean; yes enables calculation of Average
AggregateCalcOffset
Number; for DataType=struct, the offset of the element that is used for aggregate calculation
AggregateCalcDataType
Enum; for DataType=struct, the data type of the element that is used for aggregate calculation; see Data Types
AggregateCount
Boolean; yes enables calculation of Count
AggregateMax
Boolean; yes enables calculation of Maximum
AggregateMin
Boolean; yes enables calculation of Minimum
AggregateStddev
Boolean; yes enables calculation of Standard Derivation
ArrayLength
Number; number of array elements to store
DataType
Enum; data type of value to store; see Data Types; for structures, a : and the structure name is appended
ElementLength
Number; for DataType=string, the maximum string length; for DataType=struct, the data length of the structure
FileResolution
Enum; The resolution for creating new data files; see Resolution
FileSave
Boolean; yes to enable file save
MaxFileCount
Number; the maximum number of live data value files to create
MaxMem
Number; number of bytes to use for live data storage of memory variable
MinimumFreeDiskSpace
Number; mimimal required free disk space; 0 uses globally defined value; values < 100 are percent; value >= 100 are bytes
OPCActive
Boolean; yes to enable automatic data acquisation of OPC data
OPCConnName
String; OPC Connection name to use
OPCGroupName
String; OPC Connection group name to use
OPCVarName
String; OPC Variable name to use
ReadResolution
Enum; The resolution of OPC data acquisation; see Resolution

Structs.ini contents

The Structs.ini is an ini-file with multiple sections. Each section describes one data structure used in the data files. The section name is equal to the structure name, some characters with special meaning are replaced by <xx> where xx is the hex-code of the character. The sequence of section entries describes the sequence of structure elements. Valid keys are:

Header
<Structure Name>,<Flags>
<Structure Name> - the structure name (repeated, for historical reasons)
<Flags> - hexadecimal encoded flag bits; internal use only; not required for decoding
Element
<Element Name>,<Data Type>,<Data Type Flags>,<Element Flags>,<Array Size>,<Original Type>
<Element Name> - the element name; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Data Type> - data type of the structure element; see Data Types; for structures and enumerations, a : and the structure/enumeration name is appended; for strings, a : and the maximum string length is appended
<Data Type Flags> - hexadecimal encoded flag bits; internal use only; not required for decoding
<Element Flags> - hexadecimal encoded flag bits; valid values from the following list are OR'ed tokether:   0x0010 - PACKED - if set, no bit padding exists before this element
  0x0020 - PADDING - if set, this element is a padding element and can be ignored or hidden by user application
  0x0040 - VARIABLE_ARRAY_LENGTH - if set, the array length is given by another structure element instead of the ArrayLength field
  0x0080 - VARIABLE_STRING_LENGTH - if set, the string length is dynamic, instead of a fixed string length with padding
  0x0100 - OPTIONAL - if set, the element is optional; its presence is determined by another structure element
  all other bits are reserved for internal use only
<Array Size> - the array length in elememnts <Original Type> - a protocol specific "original type" for imported structs; may be safely ignored
ElementDynamic
optional; <Length Element>,<Switch Element>,<Switch Value>; this line is present only if at least one of these settings are required
<Length Element> - used if VARIABLE_ARRAY_LENGTH is set; the name of a structure element that gives the array length of this element; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Switch Element> - used if OPTIONAL is set; the name of a structure element that determines if this element is present or not; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Switch Value> - used if OPTIONAL is set; if the <Switch Element> has this value, then this element is present, otherwise it is absent
ElementComment
optional; the element comment; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.

Data file contents

The data files contain the stored values in a binary format.

File name rules: data_<resolution>_<timestamp>.bin. <resolution> is a number (0 to 7) that specifies the resolution of the data contained in this file; see Resolution. <timestamp> is the start time of this file. The timestamp format is "yyyymmddhhmm" (4 digits for year, 2 digits each for month, day, hour and minute).

All values are stored in the little-endian format (least significant byte first). Each file contains a number of records. The length of each record is determined by the settings in Var.ini.

Each entry has a 16-byte header with the following fields:

tv_sec
unsigned int64; Timestamp, given as seconds since 1970-01-01 00:00 UTC
tv_nsec
unsigned int32; Timestamp, nanoseconds in the second given by tv_sec
quality
unsigned int32; Quality for the stored value

After the header follow the user data. After the user data (without any padding) follows the next header.

Live data format (resolution 0)

These are the unmodified values as given by OPC or written by an application. Data length is given by the DataType entry in Var.ini, multiplied by the ArrayLength entry (1 if not present).

DataType=string: Each string has a 4-byte header (giving the really used number of bytes) followed by the data bytes (always ElementLength bytes).

DataType=struct: Each structure has a length as given in ElementLength

Structure encoding rules

Aggregate data format (resolutions 1 - 7)

These are aggregated values calculated during data recording. Data length is determined by the selected aggregate functions. The following fields may be present (in the given sequence):

FieldMinMaxCountAvgStdDevTypeDescription
MinimumX----DataType
AggregateCalcDataType if DataType=struct
The minimal value in the interval.
Maximum-X---DataType
AggregateCalcDataType if DataType=struct
The maximal value in the interval.
Count--X--always unsigned int64The number of data changes in the interval.
Sum---XXalways doubleThe weighted sum of all data in the interval.
tv_sec---XXalways unsigned int64The number of seconds of data recorded in the interval.
tv_nsec---XXalways unsigned int32The number of nanoseconds after the last second of data recorded in the interval.
SumSq----Xalways doubleThe sum of all squared weighted data values in the interval.

To calculate the average value in the interval: Avg = Sum / (tv_sec + tv_nsec / 1000000000)

To calculate the standard derivation in the interval: StdDev = sqrt(SumSq / (tv_sec + tv_nsec / 1000000000) - Avg * Avg)

Data types

NameSizeDescription
bit1 Bittruth value (false or true)
u8, u16, u32, u641-8 bytesunsigned integral types with 8-64 bit (1-8 bytes)
i8, i16, i32, i641-8 bytessigned integral types with 8-64 bit (1-8 bytes)
f32, f644 or 8 bytesIEEE-floating-point values with 32 or 64 bit (4 or 8 bytes)
stringvariablecharacter string, UTF-8 encoded
time12 byteUNIX timestamp, with resolution 1 nanosecond
structvariablestructured data type, encoded according to this rules
enum4 byteenumeration, always encoded as u32

Resolution values

The following resolutions are available for data acquisation:

Fastest
Save values as fast as the source generates them
Second
Save at most one value per second
Minute
Save at most one value per minute
Hour
Save at most one value per hour
Day
Save at most one value per day

The following resolutions are available for file rotation:

Hour
Start a new file at the beginning of each hour (UTC time): 2000-01-01 00:00, 2000-01-01 01:00, 2000-01-01 02:00, ...
Day
Start a new file at midnight (UTC time) of each day: 2000-01-01 00:00, 2000-01-02 00:00, 2000-01-03 00:00
Week
Start a new file at midnight (UTC time) of each monday: 2000-01-03 00:00, 2000-01-10 00:00, 2000-01-17 00:00, 00
Month
Start a new file at midnight (UTC time) of the first day of each month: 2000-01-01 00:00, 2000-02-01 00:00, 2000-03-01 00:00, ...
Year
Start a new file at midnight (UTC time) of january 1st: 2000-01-01 00:00, 2001-01-01 00:00, 2002-01-01 00:00, ...