Reading and Writing to HEC-DSS Files
Retrieving and storing data from a HEC-DSS file is done through the class HecDss in the hec.heclib.dss package. The first call that should be made is the static member HecDss.open(string filename), which will return a HecDss object to retrieve and store data, as well as perform math and other functions.
You need to import the classes that you use as well as the hec.script package. To do this, include a subset of the following lines at the beginning of your script:
import example
from hec.script import Plot, MessageBox
from hec.io import TimeSeriesContainer
from hec.io import PairedDataContainer
from hec.hecmath import TimeSeriesMath
from hec.hecmath import PairedDataMath
from hec.heclib.dss import HecDss, DSSPathname, HecDataManager
HecDss Class
from hec.heclib.dss import HecDss, DSSPathname
HecDss.open(string filename)
HecDss.open(string filename, string startTime, string endTime)
HecDss.open(string filename, string timeWindow)
HecDss.open(string filename, Boolean fileMustExist)
The HecDss class is used to gain access to an HEC-DSS file, as illustrated below. You must import HecDss from hec.heclib.dss.
Example 15: Opening and Releasing a DSS File
from hec.heclib.dss import HecDss
theFile = HecDss.open("MyFile.dss")
#or
theFile = HecDss.open("C:/temp/sample.dss", True)
#Finished using the file – release it
theFile.done()
Using one of the methods that specifies a time window affects the operation of the get() and read() methods as described in the Pattern Strings section below.
Note: The preferred manner of releasing access to an HEC-DSS file in most cases is to use the done() method and not the close() method. This is because HEC-DSS maintains information about file access and can grant access to a file released with done() much more quickly than it can to one released with close(). HEC-DSS will close files when a script exits or when they have not been accessed for a while. The only time that the close() method should be called explicitly is to perform some operating system operation on it, such as renaming or deleting the file.
HecDss Retrieve and Store Functions
HecDss objects are used to retrieve and store data sets in a DSS file, as well as several utility functions for DSS files. Data is retrieved and stored using DataContainers. DataContainers are simple classes that are database independent and are how most Hec classes exchange data. DataContainers are a base class for several data types, such as time series data, paired data, gridded data, etc. DataContainers and their related classes are described following the HecDss class. If the data set you are retrieving does not exist, the DataContainer will have the numberValues set to 0 (zero).
HecDss, and most classes accessed through scripts, use Java Exceptions for error handling. Use a try block to catch and process errors. If data cannot be stored, HecDss will throw an exception. Refer to Handling Exceptions in Scripting Basics for more information.
Tip: From the HEC-DSSVue main screen, select a pathname, right click and select Copy Pathnames to Clipboard. You can then paste the pathname into your script and avoid having to type it. You cannot use condensed pathnames for these functions.
The HecDss methods for retrieval, storage, and utility methods are described in the tables below. Time Series Data Storage Constants (Regular and Irregular) are also shown below.
HecDss Retrieval and Storage Methods
Method | Returns | Description |
get(String pathname) | DataContainer | Retrieves data for the single pathname and returns in a DataContainer. DataContainer will be a TimeSeriesContainer, PairedDataContainter, etc., depending on the data type. If a default time window has been set via one of the open() or setTimeWindow() methods, the "D" part of a time series pathname is ignored and the data for the time window is retrieved. |
get(string pathname, Boolean getEntireDataSet) | DataContainer | Ignores the "D" part of time series data and retrieves data for the entire date span in the DSS file. |
get(DSSIdentifier dssid) | DataContainer | Retrieves data for the pathname and time window given in the dssid |
put(DataContainer dataContainer) | None | Store the data in DataContainer in DSS. The DataContainer must be a TimeSeriesContainer or PairedDataContainter, etc. Throws an exception if the data cannot be stored |
HecDss Additional Primary Methods
Method | Returns | Description |
done() | None | Tells HEC-DSS that you are finished using it and releases the file. Use this instead of close(). |
getPathnameList() | list | Returns a list of the pathnames in the file. |
getTimeSeriesExtents(String pathname, HecTime start, HecTime end) | Boolean | Sets the contents of the start and end parameters to the date/time of the first and last piece of data in the entire data span. Returns whether the pathname exists. |
isOpened () | Boolean | Returns if the DSS file is accessible. |
open(String dssFileName) | HecDss | Opens the DSS file and returns a HecDss object for further access. Creates the file if it does not exist. Throws an exception if the file cannot be created or opened. |
open(String dssFileName, Boolean fileMustExist) | HecDss | Opens the DSS file and returns a HecDss object for further access. Throws an exception if you indicate the file must exist and it does not. |
recordExists(String pathname) | Boolean | Tests to see if the single record exists. |
Example 16: Reading from DSS
from javax.swing import JOptionPane
from hec.heclib.dss import HecDss
from hec.script import Plot
import java
try :
try :
myDss = HecDss.open("C:/temp/sample.dss")
flow = myDss.get("/RUSSIAN/NR UKIAH/FLOW/01MAR2006/1HOUR//")
plot = Plot.newPlot("Russian River at Ukiah")
plot.addData(flow)
plot.showPlot()
plot.stayOpen()
except Exception, e :
JOptionPane.showMessageDialog(None, ' '.join(e.args), "Python Error", JOptionPane.ERROR_MESSAGE)
except java.lang.Exception, e :
MessageBox.showError(e.getMessage(), "Java Error")
finally :
myDss.done()
The following plot is displayed when running the script above. The plot includes Marcy of 2016 because there is a date '01MAR2016' in the path.
Example 17 : Reading a complete data set from DSS
from hec.script import Plot,MessageBox
from hec.heclib.dss import HecDss
import java
try :
myDss = HecDss.open("C:/temp/sample.dss")
flow = myDss.get("/MISSISSIPPI/ST. LOUIS/FLOW//1DAY/OBS/", 1)
if flow.numberValues == 0 :
MessageBox.showError("No Data", "Error")
else :
plot = Plot.newPlot("Mississippi")
plot.addData(flow)
plot.showPlot()
except Exception, e :
MessageBox.showError(' '.join(e.args), "Python Error")
except java.lang.Exception, e :
MessageBox.showError(e.getMessage(), "Error")
finally :
myDss.done()
Example 17 results in a plot of the entire data set because the D part (date) of the path was left blank.
Example 18: Writing Time series data to DSS
from hec.script import Plot, MessageBox
from hec.heclib.dss import HecDss, HecTimeSeries
from hec.io import TimeSeriesContainer
from hec.heclib.util import HecTime
import java
try:
myDss = HecDss.open("myFile.dss")
tsc = TimeSeriesContainer()
tsc.fullName = "/Basin/loc/FLOW/01NOV2002/1Hour//"
start = HecTime("01Nov2022","0800")
tsc.interval = 60
flows = [0.0,2.0,1.0,4.0,3.0,6.0,5.0,8.0,7.0,9.0]
times =[]
for value in flows:
times.append(start.value())
start.add(tsc.interval)
tsc.times = times
tsc.values = flows
tsc.numberValues = len(flows)
tsc.units ="CFS"
tsc.type = "PER-AVER"
myDss.put(tsc)
except Exception, e:
MessageBox.showError(''.join(e.args),"Python Error")
finally:
myDss.close()
Output from the Example above
11:57:20.724 -----DSS---zopen Existing file opened, File: c:\programs\HEC-DSSVue-v3.2.1.130\myFile.dss
11:57:20.724 Handle 3; Process: 2424; DSS Versions - Software: 7-HV, File: 7-HV
11:57:20.724 Single-user advisory access mode
11:57:20.725 -----DSS--- zwrite Handle 3; Version 2: /Basin/loc/FLOW/01Nov2022/1Hour//
11:57:20.725 -----DSS---zclose Handle 3; Process: 2424; File: c:\programs\HEC-DSSVue-v3.2.1.130\myFile.dss
11:57:20.726 Number records: 1
11:57:20.726 File size: 15826 64-bit words
11:57:20.726 File size: 123 Kb; 0 Mb
11:57:20.726 Dead space: 0
11:57:20.726 Hash range: 8192
11:57:20.726 Number hash used: 1
11:57:20.726 Max paths for hash: 1
11:57:20.727 Corresponding hash: 2734
11:57:20.727 Number non unique hash: 0
11:57:20.727 Number bins used: 1
11:57:20.727 Number overflow bins: 0
11:57:20.727 Number physical reads: 9
11:57:20.727 Number physical writes: 8
11:57:20.727 Number denied locks: 0
Method | Returns | Description |
---|---|---|
close() | None | Close the DSS file. Only call this when you need the file closed (e.g. to rename it); otherwise call done() |
copyRecordsFrom(String toDSSFilename, list of strings pathnameList) | integer | Copy records from the open DSS file to the DSS file name specified in the call. |
copyRecordsFrom(String toDSSFilename, Vector pathnameList) | integer | Copy records from the open DSS file to the DSS file name specified in the call. |
copyRecordsInto(String fromDSSFilename, list of strings pathnameList) | integer | Copy records from the file specified in the call into the currently open DSS file |
HecDss Secondary Methods
Method | Returns | Description |
copyRecordsInto(string fromDSSFilename, Vector pathnameList) | integer | Copy records from the file specified in the call into the currently open DSS file |
delete(list of strings pathnameList) | integer | Deletes the records corresponding the list of pathnames |
delete(Vector pathnameList) | integer | Deletes the records corresponding the list of pathnames |
duplicateRecords(list of strings pathnameList, list of strings newPathnameList) | integer | Duplicates the records in the first list giving the names in the second list (same DSS file) |
duplicateRecords(Vector pathnameList, Vector newPathnameList) | integer | Duplicates the records in the first list giving the names in the second list (same DSS file) |
getCatalogedPathnames() | list of strings | Returns a list of all cataloged pathnames without generating a new catalog. |
getCatalogedPathnames(Boolean forceNew) | list of strings | Returns a list of all cataloged pathnames, optionally generating a new catalog first. |
getCatalogedPathnames(string pattern) | list of strings | Returns a list of all cataloged pathnames that match the specified pattern without generating a new catalog. |
getCatalogedPathnames(string pattern, Boolean forceNew) | list of strings | Returns a list of all cataloged pathnames that match the specified pattern, optionally generating a new catalog first. |
getDataManager() | CombinedDataManager | Gets the data manager for this DSS file |
getEndTime() | string | Returns the end time set |
getFilename() | string | Returns the name of the DSS file |
getStartTime() | String | Returns the start time set |
isRemote() | Boolean | Returns if the DSS file is being served on a remote machine (client/server mode) |
read(string pathname)1 | HecMath | Returns an HecMath object that holds the data set specified by pathname. If a default time window has been set via one of the open() or setTimeWindow() methods, the "D" part of a time series pathname is ignored and the data for the time window is retrieved. |
read(string pathname, string timeWindow) 1 | HecMath | Returns an HecMath object that holds the data set specified by pathname with the specified time window. |
read(string pathname, string startTime, string endTime) 1 | HecMath | Returns an HecMath object that holds the data set specified by pathname with the specified time window. |
renameRecords (list of strings pathnameList, list of strings newPathnameList) | integer | Renames the records in the first list to those in the second list. |
renameRecords (Vector pathnameList, Vector newPathnameList) | integer | Renames the records in the first list to those in the second list. |
setIrregularStoreMethod (integer storeMethod) | None | See table following |
setRegularStoreMethod (integer storeMethod) | None | See table following |
setTimeWindow(string timeWindow) | None | The default time window for this DSSFile. |
setTimeWindow(string startTime, string endTime) | None | The default time window for this DSSFile. |
setTrimMissing(Boolean trim) | None | Sets whether TimeSeriesMath objects retrieved via calls to read(…) will have missing data trimmed from the beginning and end of the time window. 2 |
write(HecMath dataset) 1 | integer | Write the data set to the DSS file. A return value of zero indicates success. |
write (TimeSeriesMath timeSeriesData, string storeMethod) | integer | See table following |
1 Currently, the read(…) and write(…) methods can operate only on time-series data, paired data, stream rating data, and text data represented by TimeSeriesMath, PairedDataMath, and StreamRatingMath TextMath objects, respectively. Other record types, such as gridded data are not yet supported by these methods.
2 By default, TimeSeriesMath objects retrieved via calls to read(…) contain data only between the first non-missing value and the last non-missing value within the specified time window. Calling setTrimMissing(False) causes the data retrieved to include all data for the specified time window, including blocks of missing values at the beginning and end of the time window.
Time Series Data Storage Constants (Regular and Irregular)
Integer | String | Description | |
---|---|---|---|
Regular | 0 | REPLACE_ALL | Overwrite every value in the existing data |
1 | REPLACE_MISSING_VALUES_ONLY | Don't overwrite non-missing values in the existing data | |
2 | REPLACE_ALL_CREATE | REPLACE_ALL + if new data has entire records of missing data, create empty records in existing data | |
3 | REPLACE_ALL_DELETE | REPLACE_ALL + if new data has entire records of missing data, delete records from existing data | |
4 | REPLACE_WITH_NON_MISSING | Overwrite every value in the existing data only if replacement is non-missing | |
Irregular | 0 | MERGE | Result for time window will be combination of existing and specified data |
1 | DELETE_INSERT | Result for time window will be specified data only |
Pattern Strings
The pattern strings used with the getCatalogedPathnames(…) methods filter the list of returned pathnames in a user-specified manner. Pattern strings can be specified in two modes: pathname mode and pathname part mode. Both modes make use of pathname part filters:
- Pathname mode = /Afilter/Bfilter/Cfilter/Dfilter/Efilter/Ffilter/
- Pathname part mode = [A=Afilter] [B=Bfilter] [C=Cfilter] [D=Dfilter] [E=Afilter] [F=Ffilter]
In pathname mode, filters must be supplied for all pathname parts. In pathname part mode, only those parts that will not match everything must be specified.
Filters are comprised of the following components:
Normal text characters. These characters are interpreted as they appear. For example, a pattern string of "B=XYZ" specifies matching every pathname that has a B-part of "XYZ".
Special characters. The special characters are comprised of the following list:
- '@' or '*' (used interchangeably). This character can be specified as the first and/or last character of a filter and specifies matching a string of zero or more characters. For example, a pattern string of "B=XYZ@" specifies matching every pathname that as a B-part that begins with "XYZ". A pattern string of "B=*XYZ" specifies matching every pathname that has a B-part that ends with "XYZ". A pattern string of “B=*XYZ*” specifies matching every pathname that "XYZ" anywhere in the B-part. Note that the @ or * character must be the first and/or last character of the filter (e.g. a pattern string of "B=ABC*XYZ" is invalid).
- '#' or '!' (used interchangeably). This character must be specified as the first character of a filter and specifies matching of every string except the remainder of the filter (e.g. it negates the remainder of the filter). A pattern string of "B=!XYZ*" specifies matching every pathname that does not have a B-part that begins with "XYZ".
No character. The absence of any character specifies an empty filter, which matches a blank pathname part. Both of the following pattern strings match every pathname that has a blank A-part
- "//*/*/*/*/*/"
Using getCatalogedPathnames(…) with a pattern string utilizes the pathname matching in the underlying DSS library. To accomplish more sophisticated pathname filtering, use one of the getCatalogedPathnames(…) methods in conjunction with python text parsing and matching utilities.
Time Windows
Dates and times used to specify time windows should not contain blank characters and should include the 4 digit year. An example date and time is "04MAR2003 1400". If a single string is used to specify the time window, the starting date and time must precede the ending date and time, for example dssFile.setTimeWindow ("04MAR2003 1400 06APR2004 0900"). A relative time may be used in the single string command, where the letter "T" represents the current time, and the days or hours can be subtracted or added to that. For example dssFile.setTimeWindow("T-14D T-2H") would specify the starting time as current time minus fourteen days and the ending time as the current time minus two hours.
Time Windows (specified by starting time and ending time) effect how the DSSFile.read(…) method operates for the various data types.
■ Time-series data. The time window supplied to DSS.open(…) or dssFile.setTimeWindow(…) specifies the default time window for all subsequent dssFile.read(…) operations. If no time window is supplied to DSS.open(…) then the default time window is undefined until dssFile.setTimeWindow(…) is called. If the default time window is undefined, then all dssFile.read(…) operations involving time-series data must specify a time window either implicitly via the D-part of the specified pathname or explicitly via the startTime and endTime parameters. The order of precedence for time windows is as follows:
● Explicit time window. Specified in dssFile.read(pathname, startTime, endTime) or dssFile.read(pathname, timeWindow). The D-part of the pathname is ignored and may be empty.
● Default time window. Specified in DSS.open(filename, startTime, endTime) or DSS.open(filename, timeWindow) or dssFile.setTimeWindow(startTime, endTime) or dssFile.setTimeWindow(timeWindow). The D-part of the pathname is ignored and may be empty. The default time window can be set to undefined by calling dssFile.setTimeWindow("", "").
● Implicit time window. Specified as the D-part of the pathname supplied to dssFile.read(pathname) when the default time window is undefined. The D-part of the pathname must not be empty.
■ Paired data. Time windows have no effect on reading paired data records.
■ Stream rating data. Stream rating data are comprised of a time-series of individual rating records for a common location. The reading of stream rating data is not affected by default time windows. The implicit time window for reading stream rating data is the entire time span covered by the rating records. An explicit time window may be supplied by using the dssFile.read(pathname, startTime, endTime) method. If an explicit time window is specified, a (possible) subset of the rating records is retrieved that
cover the specified time window. The explicit time window is interpreted as the time window containing all time-series data that is to be rated via the stream rating data. The set of rating records retrieved for an explicit time window meets the following criteria.
● The earliest record retrieved is the latest rating that is on or before the start of the time window. If no such record exists, the earliest record in the rating time-series is retrieved.
● The latest record retrieved is the earliest rating that is on or after the end of the time window. If no such record exists, the latest record in the rating time-series is retrieved.
● All rating records between the earliest and latest retrieved records are also retrieved.
■ Text data. Time windows have no effect on reading text records.