Time Series Data
Time Series Structs
Time series data is stored and retrieved using "zStructTimeSeries", a struct designed for exchaning time series data, including both regular interval and irregular interval data.
The primary components of zStructTimeSeries are:
- The pathname
- Start date / time and end date / time
- Data, the "values" array either as floats or doubles
- The time array, if the data is irregular-interval or random reporting
- Data quality array
- Notes character or integer array
- If profile data, the profile curves
Time Series Pathnames
A time series pathname has the following 6 parts:
A. Group or Basin
B. Location
C. Parameter
D. Block start date (determined by DSS)
E. Time interval (restricted to only those supported)
F. Version (user definable)
The D (date) part is generated by the DSS software according to the start date and end date, and the E (time interval) part is restricted to the allowed intervals. Generally, the pathname is formed using the function "zpathnameForm", and the E part is obtained from ztsGetEPartFromInterval.
Example: Storing and Retrieving Regular Interval Time Series Data
The following snippit of code generates regular interval time series data, writes it out to a HEC-DSS file, then reads it back in and prints it.
#include <stdio.h>
#include "heclib.h"
// Storing and Retrieving Regular Interval Time Series Data
int ExampleTimeSeries1()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
float fvalues[200];
char cdate[13], ctime[10];
int valueTime, status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Write a regular interval data set. Gen up the data
for (i=0; i<200; i++) {
fvalues[i] = (float)i;
}
tss1 = zstructTsNewRegFloats("/Basin/Location/Flow//1Hour/C Test/",
fvalues, 200, "21Jan2001", "1200", "cfs", "Inst-Val");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/01Jan2001/1Hour/C Test/");
status = ztsRetrieve(ifltab, tss2, -1, 1, 0);
if (status != STATUS_OKAY) return status;
// Print out. (Compute time for each ordinate, values are floats)
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (i=0; i<tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf ("Ordinate %d, for %s, %s, value is %f\n",
i, cdate, ctime, tss2→floatValues[i]);
valueTime += tss2->timeIntervalSeconds / tss2->timeGranularitySeconds; // Increment to next hour
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Times and Time Windows
For HEC-DSS time series data, a "dataset" is generally used, not individual records. A time series dataset contains data over a span of time and is defined by a start date/time and end date/time. HEC-DSS take dates as character strings (e.g. "09Jan1982") or Julian days since 01Jan1900 (where 01Jan1900is day one.) Times are typically defined as either seconds or minutes past midnight for that day.
In DSS version 7, if an explicit time window is not specified, it can be implied. For example, a condensed pathname's D part (e.g., "09Jul1973 - 20Sep2003") is the date span of the time window, and DSS will interpret that as the start of the day for the first date until the end of the day for the second date. If just a single date is given (e.g., "01Jan2000"), then the data for that single record will be retrieved.
If you do not know the time window for a data set, you can obtain it using the function "ztsGetDateRange". If you have a valid pathname with a valid "D" (date) part, ztsGetDateRange will search forwards and backwards until no more records are found to determine the "ends" of the time window. If you do not have a valid "D" (date) part, then you can have it to do an exhaustive search of the database, where it will search the catalog for the same pathname, except for the D part. This is a resource intensive call and should be avoided.
If you wish to read the entire dataset (all times or period of record), you can either just set the D part to an astrick "*", or set the zStructTimeSeries "boolRetrieveAllTimes" flag to 1 and DSS will automatically read all of the data. Do not call getSeriesTimeRange..
Generally data is stored as "End-Of-Period" (EOP), which corresponds to midnight being 2400 hours, or the end of the day compared to 0000 hours as the beginning of the day. If you have daily average flow, you cannot know the average flow until the day is complete (at the end of the day). Monthly average data is reported at the end of the month (the 30th or 31st); hourly data is reported at the end of the hour. One exception is when the start of a value is needed, such as the start measurement for a grid set. For those cases, the "Beginning of Period" time is given.
Some commercial programs do not account for time in this manner. For example, Microsoft Excel will generally ignore the time for daily data. The unset time defaults to "0" (zero), which can make it display as beginning of period. When importing Excel data into HEC-DSS, this time discrepancy needs to be accounted for.
Units and Type
Time series data has strings containing the data's units and type that is to be stored with it. The units are the common usage ones such as feet or cfs. There are no limitations on the spelling, capitalization or length of the units.
The type indicates how the data was measured or recorded, e.g., is it averaged over a period or is it a single measurement. The type field is limited to the following fields:
- "INST-VAL"(A value for a specific time)
- "INST-CUM"(A cumulative measurement, such as a precipitation mass curve)
- "PER-AVER"(An average over the previous time and this time)
- "PER-CUM"(Accumulation over the period, such as incremental precipitation)
The type determines how the data is plotted (e.g., stair step or curve), and how math operations are done on it. You will get very different answers from the math functions for different data types. The data type is a common source of error when math is preformed on data with programs outside of the DSS realm (e.g., Excel).
Regular-Interval Data
Regular-interval time series data is stored in "standard size" blocks whose length depends upon the time interval of the data. For example, daily time interval data are stored in blocks of one year (365 or 366 values) while monthly values are stored in blocks of ten years (120 values). If data does not exist for a portion of the full block, the missing values are set to the missing data flag UNDEFINED_FLOAT, which is actually -FLT_MAX. There are several functions that deal with missing data, such as zisMissingFloat() and zisMissingDouble.
The starting and ending times of a block correspond to standard calendar conventions. For example, for period average monthly data in the 1950's, part D (date part) of the pathname would be 01JAN1950, regardless of when the first valid data occurred (e.g., it could start in 1958). The 1960's block starts on 01JAN1960.
Average period data values are stored at the end of the period over which the data is averaged. For example, daily average values are given a time label of 2400 hours for the appropriate day and average monthly values are labeled at 2400 hours on the last day of the month. If values occur for times other than the end-of-period time, that time offset is stored in the header array. For example, if daily average flow reading's are recorded at 6:00 am (i.e., the average flow from 6:01 am of the previous day to 6:00 am of the current day), then an offset of 360 (minutes) will be stored in the header array.
Part E consists of an integer number and an alphanumeric time interval specifying the regular data interval. The valid intervals and block lengths are:
| Valid Data Intervals | Seconds In Interval | Block Length |
|---|---|---|
| "1Year" | 31536000 | One Century |
| "1Month" | 2592000 | One Decade |
"Semi-Month" | 1296000 | |
| "Tri-Month" | 864000 | |
| "1Week" | 604800 | |
| "1Day" | 86400 | One Year |
| "12Hour" | 43200 | One Month |
| "8Hour" | 28800 | |
| "6Hour" | 21600 | |
| "4Hour" | 14400 | |
| "3Hour" | 10800 | |
| "2Hour" | 7200 | |
| "1Hour" | 3600 | |
| "30Minute" | 1800 | |
| "20Minute" | 1200 | |
| "15Minute" | 900 | |
| "12Minute" | 720 | One Day |
| "10Minute" | 600 | |
| "6Minute" | 360 | |
| "5Minute" | 300 | |
| "4Minute" | 240 | |
| "3Minute" | 180 | |
| "2Minute" | 120 | |
| "1Minute" | 60 | |
| "30Second" | 30 | |
| "20Second" | 20 | |
| "15Second" | 15 | |
| "10Second" | 10 | |
| "6Second" | 6 | |
| "5Second" | 5 | |
| "4Second" | 4 | |
| "3Second" | 3 | |
| "2Second" | 2 | |
| "1Second" | 1 |
Data Compression
Data compression is automatic for regular-interval data. It is based on repeat values only, and is lossless. When a regular-interval time series dataset is stored, the number of repeat values (that follow one-after-the-other) are counted to ensure that it is worth compressing. If so, each value is provided a single bit to indicate if that position is a repeat from the previous value. Those bits are used to reconstruct the array when it is read.
Data compression can reduce some dataset to 3% of their original size, with no loss of precision and typically faster read times (there is less to read). It works best with precipitation values and missing data flags. There is no data compression for irregular-interval data, so if you can save data in regular-interval format where it makes sense, you'll have a much more efficient file.
Irregular-Interval Data
The irregular-interval time series conventions are similar to the regular-interval conventions except that an explicit date and time is stored with each data value whereas in regular-interval time series the date and time are implied by the location of the data within the block. Irregular-interval data is stored in variable length blocks while regular-interval data is stored in fixed length blocks. The block lengths are days, months, years, decades, and centuries. Irregular-interval data is NOT compressed, as each value has an explicit date/time stored with it. Storing data as irregular-interval will generally greatly increase the record size and file size, compared to regular-interval data.
The number of values that may be stored in one record is indefinite although it is prudent to choose a size that will be less than 5000. The user selects the appropriate block length. For example, if the data to be stored occurred once every 1-2 hours, a monthly block would be appropriate. If data were recorded once or twice a day, use a yearly block. One would not want to store data that occurred 10-12 or more times a day in a yearly block (about 5000 values) because that may exceed dimension limits in some programs.
All data are stored in variable length blocks that are incremented a set amount when necessary. Initial space for 100 data values is allocated and additional increments are for 50 data values unless otherwise set. (When the 101st data value is added to the record, a new record with a length of 150 values is written.)
The E parts (blocks) for irregular-interval data are:
| Block Name | Example Frequency |
|---|---|
| "Ir-Day" | 1-10 minute, 15 Seconds |
| "Ir-Month" | hourly |
| "Ir-Year" | daily, semi-daily |
| "Ir-Decade" | monthly |
| "Ir-Century" | annual |
Irregular interval data must always be in ascending order. When storing irregular interval data to an existing dataset, you can replace the data within your time window, or you can merge the two datasets together.
Pseudo-Regular Interval Data
Pseudo-regular interval data is time-series data that usually has a regular-interval, but not always. For example, one may have daily measurements at a gage, but during an event, more measurements might be reported. Or, there might be "Local time series" data, where a measurement is taken every morning at 8:00 am, regardless if daylight savings time is in effect or not (meaning some days will be fore 8:00 am, others for 7:00 am).
Pseudo-regular data is actually stored using the irregular-interval time series conventions, except the "E part" looks like a regular-interval preceded by a tilde "~". The rest of the E part following the tilde should be the standard time interval that the data is usually at, for example "~1Hour". Because pseudo-regular data follow irregular-interval conventions, there is a specific time stored with every value and the data is not compressed.
Storage of Time Series Data
Storing Regular Interval Data
Regular interval time series data only uses a start date/time and position from the start for data values; no time array is stored with the data as the date and time are implied from the position within the block. For a position where data does not exist, a missing data flag is stored (UNDEFINED_FLOAT or UNDEFINED DOUBLE), which is usually compressed when stored or trimed when read. For regular interval data you do not need to provide a time array for storing data, only the start date/time and data array. The E part of the pathname identifies the time interval.
Storing Irregular Interval Data
A date/time is explicitly stored with every data value for irregular interval time series data, so a time array is specified along with the data array in zStructTimeSeries. To specify times for irregular data, set the startDateBase parameter to the date of the first value, then geneate a int array that contains ascending times in minutes (or seconds) for each data value from the start date. There are several date and time functions to aid in this task.
ztsStore
Time series data is stored in DSS using the function "ztsStore":
int ztsStore(long long *ifltab, zStructTimeSeries *tss, int storageFlag)
Returns:
STATUS_OKAY (zero) for a successful operation
< 0 (a large negative number) if an error occurred. See the section on error processing for more information.
storageFlag:
For regular interval data:
0 Always replace data.
1 Only replace missing data.
2 Write regardless, even if all missing data (write a missing record)
3 If a record is all missing, do not write it, and delete it from disk if it exists.
4 Do not allow a missing input data to replace a valid data.
For irregular interval data, this is a flag to indicate whether to replace or merge new data with old data. Generally replace is used for editing/changing data, merge for adding data.
0 Merge. If a value at a time does not exist, that is inserted. If it does exist, it is replaced.
1 Replace. The data read from the start to end is removed and this data replaces it.
Retrieval of Time Series Data
Time series data is retrieved from DSS using a zStructTimeSeries along with a valid pathname and time window. The time window can either be implicit or explicit. An implicit time window generally indicates that you want to read all the data in a single record. For example, if you have a pathname like "/A/B/C/01Jan2005/1Day/F/", with no other time window given, that would imply to read all the data for the year 2005 (365 values). An explicit time window is specified with a start date and time, and an end date and time, or in the D part of the pathname. Another way an explicit time window can be provided is by giving the date span in the D part, similar to a "condensed catalog" pathname is. This method has the start date followed by " – " (space, dash, space) and then the end date. For example:
"/A/B/C/22Jan2005 – 17Mar2012/1Day/F/"
Will read all the data from the start of the day on 22Jan2005 through the end of the day on 17Mar2012. This method is mainly for convenience when you already have a condensed pathname; using the start time and end time in the time series container is preferred.
You must also have the exact time interval (E part) and other pathname parts. The E part also indicates if the data is regular interval or irregular interval. If you do not have a full complete pathname, or a time window, then you can either search the DSS file for pathnames that meet your criteria or set a flag in the zStructTimeSeries to read data for all of the times for that dataset.
Time series data, both regular interval and irregular interval, is retrieved using function "ztsRetrieve":
int ztsRetrieve(long long *ifltab, zStructTimeSeries *tss, int retrieveFlag,
int retrieveDoublesFlag, int boolRetrieveQualityNotes);
Returns:
STATUS_OKAY (zero) for a successful operation
< 0 (a large negative number) if an error occurred. See the section on error processing for more information.
retrieveFlag:
For regular interval data:
0 Adhere to time window provided.
-1 Trim data. Remove missing values at the beginning and end of data set (not inside)
For irregular interval data:
0 Adhere to time window provided.
1 Retrieve one value previous to start of time window.
2 Retrieve one value after end of time window.
3 Retrieve one value before and one value after time window.
retrieveDoublesFlag indicates if data should be returned as floats or doubles:
0 Retrieve as stored. If missing, will return as doubles.
1 Retrieve as floats.
2 Retrieve as doubles.
boolRetrieveQualityNotes indicates if quality and notes should be retrieved, if they exist. Do not retrieve if you do not plan on using them:
0 Do not retrieve quality and notes
1 Retrieve quality and notes, if they exist.
zStructTimeSeries
zStructTimeSeries struct objects are created by the following functions. The first two are generally for retrieving data, while the next two are for storing regular interval data and the last two are for storing irregular interval data.
When you are done with a struct, you must always free its memory with function zstructFree.
Note: These functions do not copy times and values arrays. You cannot change those arrays between struct creation and storing or retrieving.
zStructTimeSeries* zstructTsNew(const char* pathname)
zStructTimeSeries* zstructTsNewTimes(const char* pathname, const char* startDate, const char* startTime, const char* endDate, const char* endTime)
zStructTimeSeries* zstructTsNewRegFloats(const char* pathname, float *floatValues, int numberValues, const char *startDate, const char *startTime, const char *units, const char *type)
zStructTimeSeries* zstructTsNewRegDoubles(const char* pathname, double *doubleValues, int numberValues, const char *startDate, const char *startTime, const char *units, const char *type)
zStructTimeSeries* zstructTsNewIrregFloats(const char* pathname, float floatValues, int numberValues, int *itimes, int timeGranularitySeconds, const char startDateBase, const char *units, const char *type)
zStructTimeSeries* zstructTsNewIrregDoubles(const char* pathname, double doubleValues, int numberValues, int *itimes, int timeGranularitySeconds, const char startDateBase, const char *units, const char *type)
void zstructFree(void *zstruct);
zStructTimeSeries Definition
typedef struct {
// Private
int structType;
char *pathname;
// Time information
int julianBaseDate; // julianBaseDate (* timeGranularitySeconds) + times[i] is real date (often zero).
// julianBaseDate is only added to times array, not start end or other julian dates
int startJulianDate; // Time window start date and time (independent of base date)
int startTimeSeconds; // Code uses seconds, regardless of granularity
int endJulianDate; // Time window end date and time (includes missing)
int endTimeSeconds; //
// Number of seconds each unit in times array has.
// Set to zero on read for default (usually minutes). Returns with actual value
// On write, zero (default) is minutes, or set to actual value
// Note - how the times are stored is independent of this; always seconds except for ir-century
int timeGranularitySeconds;
int timeIntervalSeconds; // Informational (on return only)
int timeOffsetSeconds; // Informational (on return only)
int *times; //* Time from julianBaseDate, using timeGranularitySeconds.
// Required for irregular, ignored for regular
int boolRetrieveAllTimes; // On retrieve, expand time window to get all (times) data for this path
// Data
int numberValues; // Number of values and times. For profile, number of times.
int sizeEachValueRead; // 1 = float, 2 = double
int precision; // -1 = not set, otherwise decimal places for each value
float *floatValues; // Either float or double, not both
double *doubleValues; // The array not used must be null
char *units;
char *type;
// Pattern data, such as average daily temp, a hydrograph, etc.
// Usually has a time reference, but no specific time (e.g., day of the year)
int boolPattern;
// Profile Data - use either arrays above or these, not both
int profileDepthsNumber;
// either floats or doubles, not both! (set other to null.)
float *floatProfileDepths; // int depths[numberDepths]
float *floatProfileValues; // float[numberDepths][numberTimes];
double *doubleProfileDepths; // int depths[numberDepths]
double *doubleProfileValues; // double[numberDepths][numberTimes];
char *unitsProfileDepths;
char *unitsProfileValues;
///////////////
char *timeZoneName; // Time zone of the data (may or may not match location time zone)
// Quality and Notes (Optional)
// meant to be a pointer to something like int quality[1000][2]
int *quality;
// The length of each quality element (e.g., 2). 0 for no quality
int qualityElementSize;
int qualityArraySize; // Total int size used for retrieval only (will not be greater). Not used for storing
// Note - you cannot have both inotes and cnotes; one or the other or neither.
// (they occupy the same space). inotes are fixed length, cnotes are \0 terminated variable length.
int *inotes;
int inoteElementSize;
int inotesArraySize; // Total int size used for retrieval only (will not be greater). Not used for storing
// Character notes, one line per value, each line terminated by "\0"
char *cnotes;
int cnotesSize; // On retrieval, this must be the size of cnotes
int cnotesLengthTotal; // Must be set for storage, returns actual value on retrieval
int *userHeader;
int userHeaderSize; // Size user header array; maximum to read
int userHeaderNumber; // number read
zStructLocation *locationStruct;
// Informational only (on return only)
int dateOfFirstRecFound;
int dataType;
long long lastWrittenTime; // Seconds since 1970
long long fileLastWrittenTime;
char programName[17];
int numberAttributes;
char **attributeKeys;
char **attributes;
// Private
char *pathnameInternal; // A COPY of the pathname. This may be changed by dss functions.
// E part identifies interval.
int processedAlready; // 0 = not processed yet; 1 = processed for retrieve; 2 = processed for storage
ztsTimeWindow *timeWindow;
// knowing which variables were allocated by the ztsNew functions,
// instead of the calling program
char allocated[zSTRUCT_length];
} zStructTimeSeries;zStructTimeSeries Use for Storing
char *pathname: Required - Usually input by a zstructTsNew function. A vaild time series pathname, except the D part is genereally ignored because the date / times need to be specified.
int startJulianDate, startTimeSeconds: Required - Usually input by a zstructTsNew function. The Julian day (days since 01Jan1900) and time after midnight, in seconds, for the start of the time window or the first piece of data. The Julian day is independent (does not use) the base date. If regular interval data, this will be the date/time of the first data value. If irregular interval data and the "replace" function is used, any data between this date/time and the date/time of the first value will be removed.
int endJulianDate, endTimeSeconds: Optional - Usually input by a zstructTsNew function. The Julain day and time after midnight, in seconds, for the end of the time window or the last piece of data.
This is ignored for regular interval data (the end date/time is computed based on the number of values) and only used with irregular interval data and the "replace" function. For the replace function, any data between this date/time and the date/time of the last value will be removed.
int julianBaseDate: Usually computed by a zstructTsNew function and is usually zero. If values in the times array may overflow int values, (extended dates, or second interval), then this is the the Julain day of the first piece of data. It will be added to values in the time array to compute the date and time of each data value without overflowing an int.
int timeIntervalSeconds, timeOffsetSeconds: Not used for storing data; the E part of the pathname contains the time interval and the times imply the time offset.
int timeGranularitySeconds: Usually computed by a zstructTsNew function and is usually 1 or 60. This is the "precision" of the times used, usually 60 seconds for most data, and 1 second for data with a time interval less than one minute. If the time window extends a large time span (10,000's of years), this will be adjusted up to prevent int overflow in the time array. timeGranularitySeconds can only be 1 second, 60 seconds (1 minute), 3600 seconds (1 hour), or 86,400 seconds (1 day). The last two values are very rare.
int *times: Ignored for regular interval data and required for irregular interval data. Usually input by a zstructTsNew function. An int array containing the time of each value in the data array. Each time is in timeGranularitySeconds (usually minutes) from the julianBaseDate (usually zero). Often times are computed using functions dateToJulian and timeStringToSeconds. For example,
julian = dateToJulian("20June2015");
secs = timeStringToSeconds("11:30");
times(i) = ((julian – julianBaseDate) * 86400 / timeGranularitySeconds) + (secs / timeGranularitySeconds);
float *floatValues or double *doubleValues: Required - Usually input by a zstructTsNew function. The data to store, either as floats or doubles. The array that is not used must be null.
int numberValues: Required - Usually input by a zstructTsNew function. The number of values in the data array and times array (for irregular interval data).
char *units: Required - Usually input by a zstructTsNew function. A null terminated string containing the units (e.g., "cfs") for the data, as discussed previously.
char *type: Required - Usually input by a zstructTsNew function. A null terminated string containing the type (e.g., "INST-VAL") for the data, as discussed previously.
int precision: Optional. The display precision of the data. This is how many decimals to the right of the decimal point to show in tables, e.g. "1.234" would have a precision of 3. A value of -1 indicates not set. Note, this is only the display, the full value of the number is stored in DSS.
char *timeZoneName: Optional. An variable containing the name of the time zone for this data set. (C code only stores / retrieves time zone names; Java code will process it.)
Extended zStructTimeSeries Variables for Storing
The following are additional variables that can be stored with the record in addition to, or instead of, data defined above.
Profile Data
Profile data is a double dimensioned time series data set and can be regular interval or irregular interval. Lake – depth temperatures is a common profile data set. A profile has depths and values, dimensioned as depths[profileDepthsNumber] and values[profileDepthsNumber][numberTimes]. When profile data is used, the floatValues and doubleValues arrays must be null.
See the section on profile data for more information.
int profileDepthsNumber: Required for profile data. The number of depths in the depths array and the values array; the dimension of depths and the first dimension argument for values.
int numberValues: Required. Both the number of values and number of times for the values array.
float *floatProfileDepths or double *doubleProfileDepths: Required. The depths array providing the second parameter to the values array, either as floats or doubles. The array that is not used must be null. The array is to be dimensioned to profileDepthsNumber (floatProfileDepths[profileDepthsNumber]). An example is 0 meters, 5 meters, 10 meters, etc.
float *floatProfileValues or double *doubleProfileValues: Required. The values array, either as floats or doubles. The array that is not used must be null. The array is to be dimensioned to profileDepthsNumber, numberValues (floatProfileValues[profileDepthsNumber][ numberValues]).
char *unitsProfileDepths: Required. A null terminated string containing the units of the depths array (e.g., “meters”).
char *unitsProfileValues: Required. A null terminated string containing the units of the values array (e.g., “deg C”).
Quality and Notes
int *quality: Optional. An array of length (numberValues * qualityElementSize) containing quality flags for the values array. See the section on quality flags for more information.
int qualityElementSize: Optional. The length, in ints, of each quality flag. If set to 1, each quality flag will occupy a single int (32 bits), if set to two, each flag will occupy 2 ints (64 bit) (equivalent of a double.)
int *inotes: Optional. An array of length (numberValues * inoteElementSize) containing integer notes, essentially another array to store information for a data set. See the section on notes for more information. If integer notes are to be stored, then character notes cannot be used (they occupy the same space.)
int inoteElementSize: Optional. The length, in ints, of each integer note. If set to 1, each note will occupy a single int (32 bits), if set to two, each note will occupy 2 ints (64 bit).
int *cnotes: Optional. A character array containing character notes, a string for each value. Each character note must be null terminated and there must be one string per value, if used. See the section on notes for more information. If character notes are to be stored, then integer notes cannot be used.
int cnotesLengthTotal: Optional. The length, in bytes, of the cnotes array.
User Header
int *userHeader: Optional. An array to store any additional information needed about the data set. The user header does not generally have a one-to-one correspondence with data, but is an area to store pertinent information, often in a keyword:item format, such as "Shift:12.4".
int userHeaderNumber: Optional. The number of ints in the userHeader array to store (regardless of the data type in that array.)
zStructTimeSeries Use for Retrieving
Input Parameters:
char *pathname: Usually input by a zstructTsNew function. A vaild time series pathname.
Time Window: An implicit or explicit time window must be supplied. An explicit time window is the starting date and time and the ending date and time. Implicit time windows include: a) Retrieving the data just for that record, using the D part as the start date; b) Retrieving all data for that pathname, regardless of time (by setting boolRetrieveAllTimes to 1); c) Using a date span in the D part to identify the time window (condensed catalog style); and d) specifying a start date and time and number of values to retrieve.
int startJulianDate, startTimeSeconds: Usually input by a zstructTsNew function. The explicit start of the time window in Julian days (days since 01Jan1900) and time after midnight, in seconds. The Julian day is independent (does not use) the base date.
int endJulianDate, endTimeSeconds: Usually input by a zstructTsNew function. The explicit start of the time time window in Julian days (days since 01Jan1900) and time after midnight, in seconds.
int boolRetrieveAllTimes: Set this to 1 (one) to retrieve the entire data set (complete time span). Note, if you know the time window, it is much more efficient to provide that instead of setting this to 1.
Return Parameters:
int startJulianDate, startTimeSeconds: If the time window was implicit, this will be the start of the time window in Julian days (days since 01Jan1900) and time after midnight, in seconds. The Julian day is independent (does not use) the base date.
int endJulianDate, endTimeSeconds: If the time window was implicit, this will be the end of the time window in Julian days (days since 01Jan1900) and time after midnight, in seconds.
int julianBaseDate: The base date for the time array. Adding this value, appropriately, to the time array will give the date and times for each value. This is often the date of the first data value.
int timeIntervalSeconds: The time interval, computed from the E part. If the data is irregular, this will be -1.
int timeOffsetSeconds: For regular interval data, this will be time between the "standard time" and value time, using the start of the interval. For example, if daily data is recorded at 8:00, it will have a standard time of 24:00 and a time offset of 28,800 seconds (8 hrs * 3600 secs/hr).
int timeGranularitySeconds: The "precision" of the times, usually 60 seconds for most data, and 1 second for data with a time interval less than one minute. If the time window extends a large time span (10,000's of years), this will be adjusted up to prevent int overflow in the time array. timeGranularitySeconds can only be 1 second, 60 seconds (1 minute), 3600 seconds (1 hour), or 86,400 seconds (1 day). The last two values are very rare.
int *times: Returned with irregular interval data; not computed for regular interval data. An int array containing the time of each value in the data array. Each time is in timeGranularitySeconds (usually minutes) from the julianBaseDate. Function getDateAndTime will return a printable date and time for values in the times array. For example,
getDateAndTime(tss->times(i), tss->timeGranularitySeconds, tss->startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
float *floatValues or double *doubleValues: The retrieved data, either as floats or doubles. The array that is not used will be null.
int numberValues: The number of values retrieved.
int sizeEachValueRead: Returns "1" if the values in the database were floats, and "2" if the values were doubles. Caution – this is what the values are in the database, not those transformed to another type.
char *units: A null terminated string containing the units (e.g., "cfs").
char *type: A null terminated string containing the type (e.g., "INST-VAL").
int precision: The display precision of the data. This is how many decimals to the right of the decimal point to show in tables, e.g. "1.234" would have a precision of 3. A value of -1 indicates not set. Note, this is only the display, the full value of the number is stored in DSS.
Extended zStructTimeSeries Variables
The following are additional variables that can be retrieved with the dataset in addition to, or instead of, data defined above. Note, these are all return values and ztsRetrieve flags need to be set to retrieve some of these.
Profile Data
Profile data is a double dimensioned time series data set and can be regular interval or irregular interval. Lake – depth temperatures is a common profile data set. A profile has depths and values, dimensioned as depths[profileDepthsNumber] and values[profileDepthsNumber][numberTimes]. When profile data is used, the floatValues and doubleValues arrays will be null.
See the section on profile data for more information.
int profileDepthsNumber: The number of depths in the depths array and the values array; the dimension of depths and the first dimension argument for values.
int numberValues: Both the number of values and number of times for the values array.
float *floatProfileDepths or double *doubleProfileDepths: The depths array providing the second parameter to the values array, either as floats or doubles. The array that is not used will be null. The array is dimensioned to profileDepthsNumber (floatProfileDepths[profileDepthsNumber]). An example is 0 meters, 5 meters, 10 meters, etc.
float *floatProfileValues or double *doubleProfileValues: The values array, either as floats or doubles. The array that is not used will be null. The array is to be dimensioned to profileDepthsNumber, numberValues (floatProfileValues[profileDepthsNumber][ numberValues]).
char *unitsProfileDepths: A string containing the units of the depths array (e.g., “meters”).
char *unitsProfileValues: A string containing the units of the values array (e.g., “deg C”).
Quality and Notes
int *quality: An array of length (numberValues * qualityElementSize) containing quality flags for the values array, if stored. See the section on quality flags for more information.
int qualityElementSize: The length, in ints, of each quality flag. If set to 1, each quality flag will occupy a single int (32 bits), if set to two, each flag will occupy 2 ints (64 bit) (equivalent of a double.)
int qualityArraySize: The size, in ints, of the array quality.
int *inotes: An array containing integer notes, if stored. See the section on notes for more information. If integer notes are used, then character notes cannot be used (they occupy the same space.)
int inoteElementSize: The length, in ints, of each integer note. If set to 1, each note will occupy a single int (32 bits), if set to two, each note will occupy 2 ints (64 bit).
int inotesArraySize: The size, in ints, of the array inotes.
int *cnotes: A character array containing character notes, a string for each value. Each character note will be null terminated and there will be one string per value, if used. See the section on notes for more information. If character notes are to be stored, then integer notes cannot be used.
int cnotesSize: The size, in ints, of the array cnotes.
int cnotesLengthTotal: The length, in bytes, of the cnotes retrieved.
User Header
int *userHeader: An array containing any additional information needed about the data set. The user header does not generally have a one-to-one correspondence with data, but is an area to store pertinent information, often in a keyword:item format, such as "Shift:12.4".
int userHeaderNumber: The number of ints in the userHeader array retrieved.
Informational variables
int dataType: The numerical value that DSS uses to identify the type of data (such as irregular-interval time series data.) These types are described in file zdataTypeDescriptions.h.
char programName[17]: If set, this is the name of the program that last wrote to this record.
char *timeZoneName: An optional variable containing the name of the time zone for this data set. (C code only stores / retrieves time zone names; Java code will process it.)
long long lastWrittenTime: The time, in milliseconds since 01Jan1970 (standard computer time), when the record (or last record in the dataset) was last written. You can use this value to see if the record has been updated since a previous check.
long long fileLastWrittenTime: The time, in milliseconds since 01Jan1970, when the DSS file was last written.
int dateOfFirstRecFound: If time series data, this is the Julian day count (since 01Jan1900) of the date from the D part of the pathname for the first record read.
Example: Storage and retrieval of irregular-interval time series data
#include <stdio.h>
#include "heclib.h"
// Storage and retrieval of irregular-interval time series data
int ExampleTimeSeries2()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
double dvalues[300];
int itimes[300];
char cdate[13], ctime[10];
int status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
// If an error occured, messages will be printed to standard out
if (status != STATUS_OKAY) return status;
// Write a irregular interval data set. Gen up the data
for (i=0; i<300; i++) {
dvalues[i] = (double)i;
itimes[i] = i * 1440;
}
tss1 = zstructTsNewIrregDoubles("/Basin/Location/Flow//~1Day/C Example/",
dvalues, 300, itimes, MINUTE_GRANULARITY, "20April2012", "cfs", "Inst-Val");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNewTimes("/Basin/Location/Flow//~1Day/C Example/",
"19April2012", "2400", "01July2013", "2400");
status = ztsRetrieve(ifltab, tss2, 0, 2, 0);
if (status != STATUS_OKAY) return status;
// Print out (values returned as doubles)
for (i=0; i<tss2->numberValues; i++) {
getDateAndTime(tss2->times[i], tss2->timeGranularitySeconds, tss2→julianBaseDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf ("Ordinate %d, for %s, %s, value is %f\n",i, cdate, ctime, tss2→doubleValues[i]);
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Time Series Data with Second Granularity
As an option DSS-7 can store time series data with a granularity of one second (the default is a granularity of one minute). This means is that the smallest time interval for regular interval data is one second, and that irregular-interval data can have a granularity of one second.
For regular interval data, all E parts that contain "Seconds" automatically use a second granularity. Time arrays retrieved from such records will have a second granularity also. To store other data (including irregular interval), set the zStructTimeSeries timeGranularitySeconds = 1 prior to setting times in the struct.
As a note, with second granularity, regular ints can overflow for seconds from a date of 01Jan1900. To mitigate this, a Julian "base date" is employed. For second granularity, the base date is usually the Julian date of the D (date) part of the first pathname in the data set, and all data times are times from that base date. With minute granularity data, the base date is usually zero, so often computations can usually get away ignoring it, but the conventions say that is to be used for date computations. For second granularity data, the base date is usually not zero, and is included in the calculations for the returned time array.
For irregular-interval data, where a time array is required, generally, you should provide the date of the first piece of data as the "startDateBase", and the time array contains times from that date. DSS stores time offsets, in seconds, from the D (date) part of the pathname, except for century data, where time offsets are in minutes.
Example: Regular-interval time series data with second granularity
#include <stdio.h>
#include "heclib.h"
// Regular interval time series data with second granularity
int ExampleTimeSeries3()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
double dvalues[300];
char cdate[13], ctime[10];
int status, i;
int valueTime;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
// If an error occured, messages will be printed to standard out
if (status != STATUS_OKAY) return status;
// Write a regular interval data set. Gen up the data
for (i=0; i<300; i++) {
dvalues[i] = (double)i;
}
tss1 = zstructTsNewRegDoubles("/Basin/Location/Flow//5Seconds/Second Granularity/",
dvalues, 300, "01May2010", "0800", "cfs", "Inst-Val");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/01May2010/5Seconds/Second Granularity/");
status = ztsRetrieve(ifltab, tss2, -1, 2, 0);
if (status != STATUS_OKAY) return status;
// Print out (values returned as doubles)
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (i = 0; i<tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf("Ordinate %d, for %s, %s, value is %f\n", i, cdate, ctime, tss2→doubleValues[i]);
valueTime += tss2->timeIntervalSeconds / tss2->timeGranularitySeconds; // Increment to next time
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Example: Irregular-interval time series data with second granularity
#include <stdio.h>
#include "heclib.h"
// Irregular-interval time series data with second granularity
int ExampleTimeSeries4()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
double dvalues[300];
int itimes[300];
char cdate[13], ctime[10];
int status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
// If an error occured, messages will be printed to standard out
if (status != STATUS_OKAY) return status;
// Write a irregular interval data set. Gen up the data
for (i=0; i<300; i++) {
dvalues[i] = (double)i;
if (i % 2 == 0) {
itimes[i] = i * 60 + 10;
}
else {
itimes[i] = i * 60 - 10;
}
}
tss1 = zstructTsNewIrregDoubles("/Basin/Location/Flow//Ir-Day/Second Granularity/",
dvalues, 300, itimes, SECOND_GRANULARITY, "01May2010", "cfs", "Inst-Val");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/01May2010/Ir-Day/Second Granularity/");
status = ztsRetrieve(ifltab, tss2, 0, 2, 0);
if (status != STATUS_OKAY) return status;
// Print out (values returned as doubles)
for (i=0; i<tss2->numberValues; i++) {
getDateAndTime(tss2->times[i], tss2->timeGranularitySeconds, tss2→julianBaseDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf ("Ordinate %d, for %s, %s, value is %f\n",i, cdate, ctime, tss2→doubleValues[i]);
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Time Series Data with Extended Date Range
DSS-7, and the underlining libraries, allow the use of large dates, typically used in statistical analyses. It has been tested for years from –5,873,711 to 5,879,610, i.e., 01Jan-5873711 to 01Jan5879610 (that's over 11 million years). DSS-6 has a limit of from year 1,000 to year 4,800.
To accommodate these dates, the int timeGranularitySeconds in zStructTimeSeries specifies what values in the zStructTimeSeries times array represent. For regular size data sets, timeGranularitySeconds will match the set time granularity (although not guaranteed), which is 60 seconds for most datasets. For large time span datasets that contain multiple records, timeGranularitySeconds may be returned as 3600 seconds (one hour) or 86400 seconds (one day). This means that the times in the times array represent hours or days, not the default of minutes, so that those times do not overflow ints in the time array. (This is not to be confused with the time granularity of how the times are actually stored in DSS (only minutes or seconds)). This, along with the Julian base date, allow large ranges of dates for time series data.
For irregular-interval data, you should provide the date of the first piece of data as the "startDateBase", and a time array which contains times from that date. For normal time spans, MINUTE_GRANULARITY is usually used. For example a time span of 01Jan9000 to 01Jan10000 would have a startDateBase of 01Jan9000 and a time granularity of MINUTE_GRANULARITY. A time span from 01Jan-9000 to 01Jan10000 requires a time granularity of at least HOUR_GRANULARITY or even DAY_GRANULARITY to prevent overflows in the times array. For regular interval data, times are implied, and not stored, so the granularity show reflect the accuracy of the time.
When using years less than 1000 with the HEC date functions, keep the year to at least 4 digits with leading zeros (if needed). For example, the year 20 may have a date of "01Jan0020", and the year -5 would be "01Jan-005". All dates are based on a day count and do not include the 1500 era Gegorian Calendar changes.
When using extended datasets, be careful not to exceed memory. There are no established limits for DSS, but programs have a finite amount of memory.
An example of storing and reading an extended dateset follows.
Example: Irregular-interval time series data over 50,000 years
#include <stdio.h>
#include "heclib.h"
// Extended Dates; Irregular-interval time series data over 50,000 years.
// Note, one could also store several smaller sets of data over the same time span
// Be careful about exceeding memory. Run this only with x64.
int ExampleTimeSeries5()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
float *values;
int *itimes;
char cdate[13], ctime[10];
int status, i;
int number;
int numberValues = 2000000;
// See if we can allocate this amount of memory
values = (float*)malloc(numberValues * 4);
if (!values) {
printf("Excceed values memory for ExampleTimeSeries5\n");
return -1;
}
itimes = (int*)malloc(numberValues * 4);
if (!itimes) {
printf("Excceed itimes memory for ExampleTimeSeries5\n");
free(values);
return -1;
}
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
// If an error occured, messages will be printed to standard out
if (status != STATUS_OKAY) {
free(values);
free(itimes);
return status;
}
// Gen up the data
// We are going to use a base date of 19Jan0201 and a daily time granularity
// (applies to times array only)
values[0] = 1.0;
itimes[0] = 0; // This will be days from base date (so first date will be 19Jan0201)
for (i=1; i<numberValues; i++) {
values[i] = (float)i + (float)1.0;
itimes[i] = itimes[i - 1] + 10; // Add 10 days
}
tss1 = zstructTsNewIrregFloats("/Basin/Location/Flow//Ir-Century/Extended dates example/",
values, numberValues, itimes, DAY_GRANULARITY, "19Jan0201", "cfs", "Inst-Val");
if (!tss1) {
printf("Excceed tss1 memory for ExampleTimeSeries5\n");
return -1;
}
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1); // Free up memory, so we do not exceed it on the read.
// You cannot free (or change) these arrays until you are finished with the store.
free(values);
free(itimes);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow//Ir-Century/Extended dates example/");
tss2->boolRetrieveAllTimes = 1; // Read the entire dataset.
status = ztsRetrieve(ifltab, tss2, 0, 0, 0);
if (status != STATUS_OKAY) return status;
if (!tss2->floatValues) {
printf("Error retrieving tss2->floatValues for ExampleTimeSeries5\n");
zstructFree(tss2);
return -1;
}
// Print out first 10 and last 10
printf("\n\n");
for (i=0; i<10; i++) {
getDateAndTime(tss2->times[i], tss2->timeGranularitySeconds, tss2→julianBaseDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf ("Ordinate %d, for %s, %s, value is %f\n",i, cdate, ctime, tss2→floatValues[i]);
}
printf("\n\n");
number = tss2->numberValues - 10;
for (i = number; i<tss2->numberValues; i++) {
getDateAndTime(tss2->times[i], tss2->timeGranularitySeconds, tss2→julianBaseDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf("Ordinate %d, for %s, %s, value is %f\n", i, cdate, ctime, tss2→floatValues[i]);
}
printf("\n\n");
zstructFree(tss2);
zclose(ifltab);
return 0;
}Ordinate 0, for 18Jan0201, 00:00:01, value is 1.000000
Ordinate 1, for 28Jan0201, 00:00:01, value is 2.000000
Ordinate 2, for 07Feb0201, 00:00:01, value is 3.000000
Ordinate 3, for 17Feb0201, 00:00:01, value is 4.000000
Ordinate 4, for 27Feb0201, 00:00:01, value is 5.000000
Ordinate 5, for 09Mar0201, 00:00:01, value is 6.000000
Ordinate 6, for 19Mar0201, 00:00:01, value is 7.000000
Ordinate 7, for 29Mar0201, 00:00:01, value is 8.000000
Ordinate 8, for 08Apr0201, 00:00:01, value is 9.000000
Ordinate 9, for 18Apr0201, 00:00:01, value is 10.000000
Ordinate 1999990, for 30Nov54958, 00:00:01, value is 1999991.000000
Ordinate 1999991, for 10Dec54958, 00:00:01, value is 1999992.000000
Ordinate 1999992, for 20Dec54958, 00:00:01, value is 1999993.000000
Ordinate 1999993, for 30Dec54958, 00:00:01, value is 1999994.000000
Ordinate 1999994, for 09Jan54959, 00:00:01, value is 1999995.000000
Ordinate 1999995, for 19Jan54959, 00:00:01, value is 1999996.000000
Ordinate 1999996, for 29Jan54959, 00:00:01, value is 1999997.000000
Ordinate 1999997, for 08Feb54959, 00:00:01, value is 1999998.000000
Ordinate 1999998, for 18Feb54959, 00:00:01, value is 1999999.000000
Ordinate 1999999, for 28Feb54959, 00:00:01, value is 2000000.000000
Quality Flags
In version 6 of HEC-DSS, you can optionally store a single 32-bit integer with each data value, which usually is reserved for a data quality flag. In version 7, the quality flag is not limited to a single 32-bit integer, but you specify the number of integers per value. (Although you can have more than one integer for a quality flag, older programs may not be compatible with this.)
Quality flag ints are in zStructTimeSeries as the array *quality, which should be dimensioned for storage as int quality[numberData][qualityElementSize], for example quality[744][2]. If quality is specified (not null) in zStructTimeSeries, they will be automatically stored.
To retrieve quality flags, set the last parameter (fifth parameter) to “1” when calling function ztsRetrieve. If quality flags exist for the records retrieve, the quality array will not be null, but will be returned as quality[numberData][qualityElementSize].
Quality Bit Settings
Single int Quality flag conventions were adopted in 1980 (for program DATCHK) and set bits as follows (where 1 is first bit, often referred to as zero):
1: Screened - set when original data has been screened
Quality of original data:
2: Okay
3: Missing
4: Questionable
5: Reject
6-7: Range of current data- an integer in [0,3]
8: Current value is different from original value
9-11: When set current value- an integer in [0,7]
0 original value, no revision
1 DATCHK
2 DATVUE
3 manual entry in DATVUE
4 original value accepted in DATVUE
12-15: Replacement method- an integer in [0, 15]
0 no revision
1 linear interpolation
2 manual change
3 replace with missing value
Tests Failed:
16: absolute magnitude
17: constant value
18: rate-of-change
19: relative magnitude
20: duration-magnitude
21: negative incremental value
22: not defined
23: gage list
24: not defined
25: user-defined test
26: distribution test
32: protect value from automatically being changed.
Data Quality Rules:
- Unless the Screened bit is set, no other bits can be set.
- Unused bits (22, 24, 27-31, 32+) must be reset (zero).
- The Okay, Missing, Questioned and Rejected bits are mutually exclusive.
- No replacement cause or replacement method bits can be set unless the changed (different) bit is also set, and if the changed (different) bit is set, one of the cause bits and one of the replacement method bits must be set.
- Replacement Cause integer is in range 0..4.
- Replacement Method integer is in range 0..4
- The Test Failed bits are not mutually exclusive (multiple tests can be marked as failed).
Bit Mappings
3 2 1
2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1
P - - - - - T T T T T T T T T T T M M M M C C C D R R V V V V S
| <---------------> <> <> | <> <+-> |
| | | | | | | +------Screened T/F
| | | | | | +-----------Validity Flags
| | | | | +--------------Value Range Integer
| | | | +-------------------Different T/F
| | | +---------------Replacement Cause Integer
| | +---------------------Replacement Method Integer
| +-------------------------------------------Test Failed Flags
+-------------------------------------------------------------------Protected T/F
Example: Regular-interval time series data with single int Quality flags
#include <stdio.h>
#include "heclib.h"
// Storing and Retrieving Regular Interval Time Series Data with quality flags
int ExampleTimeSeries6()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
float fvalues[200];
int quality[200];
char cdate[13], ctime[10];
int valueTime, status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Gen up the data
for (i = 0; i < 200; i++) {
fvalues[i] = (float)i;
// set evens to 3, odds to 5
if ((i & 1) == 0) {
quality[i] = 3;
}
else {
quality[i] = 5;
}
}
tss1 = zstructTsNewRegFloats("/Basin/Location/Flow//1Hour/C test with Quality/", fvalues,
200, "05Jan2001", "1200", "cfs", "Inst-Val");
tss1->quality = quality;
tss1->qualityElementSize = 1;
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/01Jan2001/1Hour/C test with Quality");
status = ztsRetrieve(ifltab, tss2, -1, 1, 1);
if (status != STATUS_OKAY) return status;
// Print out. (Compute time for each ordinate, values are floats)
// Assume quality flags is a single int (32 bit). We'd have a different
// print statment for quality flags > single ints.
if (tss2->quality && (tss2->qualityElementSize == 1)) {
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (i = 0; i < tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf("Ordinate %d, for %s, %s, value is %f, quality is %d\n",
i, cdate, ctime, tss2->floatValues[i], quality[i]);
valueTime += tss2->timeIntervalSeconds / tss2->timeGranularitySeconds; // Increment to next hour
}
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Time Series Data with Notes
In addition to quality, you can also store a separate integer notes array or character stings with each data value for version 7. If you store integer notes, the length of each note array must be the same for all values. If you store character strings, the strings can be of varying length, null terminated in the C code (but not in the Java code.) Typically, character strings are stored instead of integer notes, and they usually describe some attributes about the data. There are no length restrictions for quality or notes.
An example value for temperature using quality and notes might be:
72.4 3 “Sky partly cloudy with SW breeze”
Integer notes are passed in zStructTimeSeries inotes[][] and follow the same rules as quality flags. Character strings are passed in zStructTimeSeries cnotes[], and each string (for each value) must be null terminated. (You can have either integer notes or character notes, not both.) You must have a one-to-one correspondence for each data value and each note string. You can have empty note strings, which would just be a null character.
If you want to store integer notes, set zStructTimeSeries inotes[numberData][inoteElementSize], for example quality[744][2]. If inotes are specified (not null) in zStructTimeSeries, they will be automatically stored.
If you want to store character notes, pass them in zStructTimeSeries cnotes[cnotesLengthTotal] and set cnotesLengthTotal to the length of the cnotes array.
To retrieve notes, set the last parameter (fifth parameter) to “1” when calling function ztsRetrieve. If integer notes exist for the records retrieve, the inotes array will be returned as inotes[numberData][inoteElementSize] and cnotes will be null. If character notes exist, then cnotes will be returned as cnotes[cnotesLengthTotal] and inotes will be null.
Example: Time series data with multiple quality flags and character note strings
#include <stdio.h>
#include <string.h>
#include "heclib.h"
// Regular interval time series data with multiple int quality flags and character notes
int ExampleTimeSeries7()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
float fvalues[200];
int quality[800]; // quality[200][4]; (but C is not very double array friendly)
char cdate[13], ctime[10];
int valueTime, status, i, j, n;
int cnotesPos;
char alpha[] = "abcdefghijklmnopqrstuvwxyz";
int maxNotes = 5000;
char cnotes[5000];
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Gen up the data
cnotesPos = 0;
for (i = 0; i < 200; i++) {
fvalues[i] = (float)i;
n = i * 4;
for (j = 0; j < 4; j++) {
quality[n+j] = (i * 10) + j;
}
n = i / 25;
n = i - (n * 25) + 1;
for (j = 0; j < n; j++) {
if (cnotesPos < maxNotes) {
cnotes[cnotesPos++] = alpha[j];
}
}
if (cnotesPos < maxNotes) {
cnotes[cnotesPos++] = '\0';
}
}
tss1 = zstructTsNewRegFloats("/Basin/Location/Flow//1Hour/C Quality and Notes/", fvalues,
200, "20Jan2010", "2400", "cfs", "Inst-Val");
tss1->quality = quality;
tss1->qualityElementSize = 4;
tss1->cnotes = cnotes;
tss1->cnotesLengthTotal = cnotesPos;
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/01Jan2010/1Hour/C Quality and Notes/");
status = ztsRetrieve(ifltab, tss2, -1, 1, 1);
if (status != STATUS_OKAY) return status;
// Print out. (Compute time for each ordinate, values are floats)
if (tss2->quality && tss2->cnotes) {
cnotesPos = 0;
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (i = 0; i < tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
printf("Ordinate %d, for %s, %s, value is %f\n", i, cdate, ctime, tss2→floatValues[i]);
printf(" Quality: ");
n = i * tss2→qualityElementSize;
for (j = 0; j < tss2->qualityElementSize; j++) {
printf(" %d ", tss2->quality[n + j]);
}
printf("\n");
printf(" Cnotes: %s\n", &tss2→cnotes[cnotesPos]);
cnotesPos += (int)strlen(&tss2->cnotes[cnotesPos]) + 1;
valueTime += tss2->timeIntervalSeconds / tss2->timeGranularitySeconds; // Increment to next hour
}
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Ordinate 0, for 20Jan2010, 2400, value is 0.000000
Quality: 0 1 2 3
Cnotes: a
Ordinate 1, for 21Jan2010, 0100, value is 1.000000
Quality: 10 11 12 13
Cnotes: ab
Ordinate 2, for 21Jan2010, 0200, value is 2.000000
Quality: 20 21 22 23
Cnotes: abc
Ordinate 3, for 21Jan2010, 0300, value is 3.000000
Quality: 30 31 32 33
Cnotes: abcd
Ordinate 4, for 21Jan2010, 0400, value is 4.000000
Quality: 40 41 42 43
Cnotes: abcde
Ordinate 5, for 21Jan2010, 0500, value is 5.000000
Quality: 50 51 52 53
Cnotes: abcdef
Ordinate 6, for 21Jan2010, 0600, value is 6.000000
Quality: 60 61 62 63
Cnotes: abcdefg
...
Time Series "Profile" Data
“Profile” data allows the storage of several values for each time. It is essentially a marriage of time series data and paired data. It is called “Profile” because its most common use is a time series temperature profile for lakes. For profile data, there is an independent variable (depths) and a set of dependent variables (temperatures at those depths over time.) The independent variable (depths) describes the dependent variables and is NOT associated with a time. The C part must contain the independent variable name followed by a dash (“-“) and then the independent variable name. For lake profile, this would be “/Depth-Temperature/”.
The number of independent variables must be consistent for the entire data set; for example, if you have 10 depth readings, you must have 10 depth readings for the entire data set (although some may be marked as missing). You must have a independent variable set, even if not used. If you wanted to store multiple time series values, you would still need to have a independent variable array, although it may contain all zeros.
Profile data may have quality and notes, as other time series conventions. Profile data may be regular-interval or irregular-interval; minute (default) granularity or second granularity.
Profile depths are equivalent to columns in paired data, and times are equivalent to rows. You need to store rows x columns number of data; use a missing flag if you do not have a value for any positions in the array. The rows are the first part of the dimension, and columns the second, so the dimension in zStructTimeSeries is floatProfileValues or doubleProfileValues [numberTimes][numberDepths];. Because C is not very double array friendly, we often just compute the position as (time ordinate * numberDepths) + depth number.
Profile variable names in zStructTimeSeries follow lake temperature profiles for clarity (numberDepths is clearer than number independent variables). Units are stored for both independent and dependent variables.
Example: Regular-interval time series profile data
#include <stdio.h>
#include <math.h>
#include "heclib.h"
// Regular-interval time series profile data
int ExampleTimeSeries8()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
double profileDepths[20];
double profileValues[1000][20];
int numberDepths = 20;
int numberValues = 1000;
char cdate[13], ctime[10];
int valueTime, status, i, j, n;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Gen up the data
for (i = 0; i<numberDepths; i++) {
profileDepths[i] = (double)(i * 3);
}
for (i = 0; i<numberValues; i++) {
for (j = 0; j<numberDepths; j++) {
profileValues[i][j] = 10.0 * (sin((double)i) + sin((double)j) + 1.0);
}
}
// There is no function to create a profile struct, so
// we will add all the parameters manually.
tss1 = zstructTsNew("/Basin/Location/Depth-Temperature//1Day/C Profile Example/");
tss1->startJulianDate = dateToJulian("20Jan2010");
tss1->startTimeSeconds = timeStringToSeconds("1200");
tss1->unitsProfileDepths = "Meters";
tss1->unitsProfileValues = "deg C";
tss1->type = "Inst-val";
tss1->profileDepthsNumber = numberDepths; // 20
tss1->numberValues = numberValues; // 1000
tss1->doubleProfileDepths = profileDepths;
tss1->doubleProfileValues = &profileValues[0][0];
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Depth-Temperature/01Jan2010/1Day/C Profile Example/");
tss2->boolRetrieveAllTimes = 1;
status = ztsRetrieve(ifltab, tss2, -1, 2, 0);
if (status != STATUS_OKAY) return status;
// Print out.
for (int i = 0; i<tss2->profileDepthsNumber; i++) {
printf("Ordinate %d, depth: %f\n", i, tss2→doubleProfileDepths[i]);
}
printf("\nValues:\n");
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (int i = 0; i<tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
n = i * tss2→profileDepthsNumber;
printf("Ordinate %d, for %s, %s, First temp: %f, second: %f, third: %f\n",
i, cdate, ctime, tss2→doubleProfileValues[n],
tss2->doubleProfileValues[n+1], tss2→doubleProfileValues[n+2]);
valueTime += tss2->timeIntervalSeconds / tss2→timeGranularitySeconds;
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Ordinate 0, depth: 0.000000
Ordinate 1, depth: 3.000000
Ordinate 2, depth: 6.000000
Ordinate 3, depth: 9.000000
Ordinate 4, depth: 12.000000
Ordinate 5, depth: 15.000000
Ordinate 6, depth: 18.000000
Ordinate 7, depth: 21.000000
Ordinate 8, depth: 24.000000
Ordinate 9, depth: 27.000000
Ordinate 10, depth: 30.000000
Ordinate 11, depth: 33.000000
Ordinate 12, depth: 36.000000
Ordinate 13, depth: 39.000000
Ordinate 14, depth: 42.000000
Ordinate 15, depth: 45.000000
Ordinate 16, depth: 48.000000
Ordinate 17, depth: 51.000000
Ordinate 18, depth: 54.000000
Ordinate 19, depth: 57.000000
Values:
Ordinate 0, for 20Jan2010, 1200, First temp: 10.000000, second: 18.414710, third: 19.092974
Ordinate 1, for 21Jan2010, 1200, First temp: 18.414710, second: 26.829420, third: 27.507684
Ordinate 2, for 22Jan2010, 1200, First temp: 19.092974, second: 27.507684, third: 28.185949
Ordinate 3, for 23Jan2010, 1200, First temp: 11.411200, second: 19.825910, third: 20.504174
Ordinate 4, for 24Jan2010, 1200, First temp: 2.431975, second: 10.846685, third: 11.524949
Ordinate 5, for 25Jan2010, 1200, First temp: 0.410757, second: 8.825467, third: 9.503732
Ordinate 6, for 26Jan2010, 1200, First temp: 7.205845, second: 15.620555, third: 16.298819
Ordinate 7, for 27Jan2010, 1200, First temp: 16.569866, second: 24.984576, third: 25.662840
Ordinate 8, for 28Jan2010, 1200, First temp: 19.893582, second: 28.308292, third: 28.986557
Ordinate 9, for 29Jan2010, 1200, First temp: 14.121185, second: 22.535895, third: 23.214159
Ordinate 10, for 30Jan2010, 1200, First temp: 4.559789, second: 12.974499, third: 13.652763
Ordinate 11, for 31Jan2010, 1200, First temp: 0.000098, second: 8.414808, third: 9.093072
...
Time Pattern
HEC-DSS provides for a type of time series data that is not associated with a specific time, but rather a sequence that may or may not be associated with relative times. This includes, for example, daily average temperatures, which are associated with a specific day of the year, but not a specific year. Also included are unit hyrdographs, which are a sequqnce of flows in time, but do not include a specific time associated with the data.
These different time sequences are called "Time Patterns" in HEC-DSS and follow all HEC-DSS time series conventions, except the "D" part of the pathname is "TS-Pattern". Time pattern data may be regular interval or irregular interval; it may have a specified period (such as one year), or just a number of values from an unspecified start (such as 120 hourly values for a unit hydrograph).
The "E"part of time pattern pathnames still contains the time interval or block length; for example the E part for daily average temperatures would be "1Day", and the record would contain 365 values. An hourly unit hydrograph would have an E part of "1Hour" and the length of that record would be the entire data set for that hydrograph. Other types of time pattern data may be irregular interval, and a block length chosen that would encompass the entire data set (such as "IR-Month" for a non-uniform hydrograph).
When you use time pattern data, you do not set a start or end time; all data is assumed to be contained in one record, which is fully defined by the pathname. When you time pattern with irregular interval data, the time time array is to be given in minutes (or seconds) from the first point (so, hourly stored as irregular pattern would be "0, 60, 120, 180, …").
Time pattern data resembles paired data, where the ordinate is a time value, but is stored using time series conventions. It is recognized that some time pattern data, such as unit hydrographs, may also be stored using paired data convention. However, paired data is a different data type and most time series functions (e.g., math functions) will not work with it, whereas they will work with pattern data. Also, sometimes time pattern data is stored for the year 3000, which is often used for stastical values. Indeed, time pattern data is usually plotted using the year 3000 for the time axis. It is up to the user's descrition choose the most approiate storage.
Example: Time Pattern with Regular Interval
#include <stdio.h>
#include <math.h>
#include "heclib.h"
// Time Pattern for Regular-interval time series data
int ExampleTimePattern1()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
float values[365];
char cdate[13], ctime[10];
int valueTime, status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Gen up the data
for (i = 0; i<365; i++) {
values[i] = (float)(10.0 * (sin((double)i) + 1.0) + 50.0);
}
// The D and E part identifies the pattern type. No date or times are given.
tss1 = zstructTsNewRegFloats("/Basin/Location/Temperature-Average/TS-Pattern/1Day/C Pattern Example/",
values, 365, "", "", "deg F", "Per-Aver");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Temperature-Average/TS-Pattern/1Day/C Pattern Example/");
status = ztsRetrieve(ifltab, tss2, -1, 1, 0);
if (status != STATUS_OKAY) return status;
// Print out. (Compute time for each ordinate, values are floats)
valueTime = tss2->startTimeSeconds / tss2→timeGranularitySeconds;
for (i = 0; i<tss2->numberValues; i++) {
getDateAndTime(valueTime, tss2->timeGranularitySeconds, tss2→startJulianDate,
cdate, sizeof(cdate), ctime, sizeof(ctime));
// The year defaults to 3000 (e.g., 01Jan3000). Remove the year part of the date.
cdate[5] = '\0';
printf("Ordinate %d, for %s, %s, value is %f\n", i, cdate, ctime, tss2→floatValues[i]);
valueTime += tss2->timeIntervalSeconds / tss2->timeGranularitySeconds; // Increment time
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Ordinate 0, for 01Jan, 2400, value is 60.000000
Ordinate 1, for 02Jan, 2400, value is 68.414711
Ordinate 2, for 03Jan, 2400, value is 69.092972
Ordinate 3, for 04Jan, 2400, value is 61.411201
Ordinate 4, for 05Jan, 2400, value is 52.431976
Ordinate 5, for 06Jan, 2400, value is 50.410759
Ordinate 6, for 07Jan, 2400, value is 57.205845
Ordinate 7, for 08Jan, 2400, value is 66.569862
…
Example: Unit Hydrograph Time Pattern with Irregular Interval
#include <stdio.h>
#include <math.h>
#include "heclib.h"
// A time pattern example for a unit hydrograph with
// irregular-interval time series data
int ExampleTimePattern2()
{
long long ifltab[250];
zStructTimeSeries *tss1, *tss2;
double values[200];
int times[200];
char ctime[10];
int day;
int timeOfDay;
int status, i;
// Open the DSS file; Create if it doesn't exist
status = zopen(ifltab, "C:/temp/Example7.dss");
if (status != STATUS_OKAY) return status;
// Gen up the data
for (i = 0; i<100; i++) {
values[i] = (double)i * 5;
times[i] = (i + 1) * 60;
}
for (i = 100; i<200; i++) {
values[i] = 500.0 + (double)(100 - i) * 5.0;
times[i] = (i + 1) * 60;
}
// No absolute times are specified, just relative
tss1 = zstructTsNewIrregDoubles("/Basin/Location/Flow/TS-Pattern/Ir-Month/C Unit Hydrograph/",
values, 200, times, MINUTE_GRANULARITY, "", "deg F", "Per-Aver");
status = ztsStore(ifltab, tss1, 0);
zstructFree(tss1);
if (status != STATUS_OKAY) return status;
// Read the data back in
tss2 = zstructTsNew("/Basin/Location/Flow/TS-Pattern/Ir-Month/C Unit Hydrograph/");
status = ztsRetrieve(ifltab, tss2, 0, 2, 0);
if (status != STATUS_OKAY) return status;
// Print out.
for (i = 0; i<tss2->numberValues; i++) {
day = 1 + (tss2->times[i] / tss2->timeGranularitySeconds) / 24;
timeOfDay = tss2->times[i] - ((day - 1) * 24 * tss2→timeGranularitySeconds);
if (timeOfDay == 0) {
// Report EOP (2400 hours, not 0000)
day–;
timeOfDay = 1440;
}
minutesToHourMin(timeOfDay, ctime, sizeof(ctime));
printf("Ordinate %d, day %d, time %s, value is %f\n", i, day, ctime, tss2→doubleValues[i]);
}
zstructFree(tss2);
zclose(ifltab);
return 0;
}Ordinate 0, day 1, time 0100, value is 0.000000
Ordinate 1, day 1, time 0200, value is 5.000000
Ordinate 2, day 1, time 0300, value is 10.000000
Ordinate 3, day 1, time 0400, value is 15.000000
Ordinate 4, day 1, time 0500, value is 20.000000
Ordinate 5, day 1, time 0600, value is 25.000000
Ordinate 6, day 1, time 0700, value is 30.000000
Ordinate 7, day 1, time 0800, value is 35.000000
Ordinate 8, day 1, time 0900, value is 40.000000
Ordinate 9, day 1, time 1000, value is 45.000000
Ordinate 10, day 1, time 1100, value is 50.000000
Ordinate 11, day 1, time 1200, value is 55.000000
Time Series Utility Functions
HEC-DSS has several utility functions for time series data. Any time searching is used (as opposed to just reading), more resources are needed. So use the following functions wisely – don't call if you don't really need to. It is more efficient to read data, including data that doesn't exist and will be returned missing, than to search for it first.
zgetRecordSize
Returns a struct with for a single DSS record (efficient).
int zgetRecordSize(long long *ifltab, zStructRecordSize *recordSize);
ztsGetDateRange
Returns the start and end dates for a full dataset, not just a single record. (requires more resources)
int ztsGetDateRange(long long *ifltab, const char *pathname, int boolFullSet,
int *firstValidJulian, int *lastValidJulian);
ztsGetInterval
Returns the time interval, in seconds, for a version 7 regular interval time series pathname. If version 6, the time interval is in minutes. If not regular interval time series, zero is returned.
int ztsGetInterval(int dssVersion, const char *pathname)