Introduction


General Concepts for HEC-DSS

HEC-DSS uses a block of sequential data as the basic unit of storage.  This concept results in a more efficient access of time series or other sequentially related data.  Each block contains a series of values of a single variable over a time span appropriate for most applications.  The basic concept underlying HEC-DSS is the organization of data into records of continuous, applications-related elements as opposed to individually addressable data items.  This approach is more efficient for scientific applications than a conventional database system because it avoids the processing and storage overhead required to assemble an equivalent record from a conventional system.

Data is stored in blocks, or records, within a file and each record is identified by a unique name called a "pathname.”  Each time data is stored or retrieved from the file, its pathname must be given.  The data and information about it (e.g., units) is stored in a "header array.”  HEC-DSS automatically stores the name of the program writing the data, the number of times the data has been written to, and the last written date and time.  HEC-DSS documents stored data completely via information contained in the pathname and stored in the header so no additional information is required to identify it.  The self-documenting nature of the database allows information to be recognized and understood months or years after it was stored.

The pathname is the key to the data’s location in the database. HEC-DSS analyzes each pathname to determine a “hash” index number. This index determines where the data set is stored within the database. The design ensures that very few disk accesses are made to retrieve or store data sets. One data set is not directly related to another so there is no need to update other areas of the database when a new data set is stored.

Because of the self-documenting nature of the pathname and the conventions adopted, there is no need for a data dictionary or data definition file as required with other database systems.  In fact, there are no database creation tasks or any database setup.  Both HEC-DSS utility programs and applications that use HEC-DSS will generate and configure HEC-DSS database files automatically.  There is no pre-allocation of space; the software automatically expands the file size as needed.

A HEC-DSS database file has a user-specified conventional name with an extension of “.dss.”  As many database files as desired may be generated and there are no size limitations, apart from available disk space.  Corps offices have HEC-DSS files that range from a few data sets to thousands.  HEC-DSS adjusts internal tables and hash algorithms to match the database size so as to access both small and very large databases efficiently. 

 

HEC-DSS database files are “direct-access” binary files with no published format.  Only programs linked with the HEC-DSS software library can be used to access HEC-DSS files.  Direct access files allow efficient retrieval and storage of blocks of data compared to sequential files.

 

A principal feature of HEC-DSS is that many users can read and write data to a single database at the same time.  This multi-user access capability is implemented with system record locking and flushing functions.  There is no daemon or other background program managing accesses to a database.  A database may exist on a Windows or Unix server machine, which can be accessed by users on PC’s or other computers via NFS or the Microsoft network, as long as locking and flushing functions are implemented.

More:

Pathnames

Catalogs

Data Conventions

Time Series Conventions