By: Karl Tarbet

HEC-DSS Background

The Hydrologic Engineering Center's Data Storage System, or HEC-DSS, is a software library that allows efficient storage and sharing of water resources related data. HEC-DSS is integral to HEC's mission and has been for nearly 50 years. Over the years it has continued to evolved; between 1978 and the early 1980s it moved from punch cards to floppy disks. Then starting in 2015, features were programmed in the C language instead of Fortran. It took ten years of effort to move from HEC-DSS 6 to HEC-DSS 7, and customer support for HEC-DSS 6 ended in July 2024.

HEC-DSS files are used extensively with HEC software products, with dozens of developers and thousands of end users relying on HEC-DSS technology every day.  HEC-DSS storage technology continues to serve one of HEC's original goals of "efficiently storing and retrieve scientific data that is typically sequential".  However, the original definition of DSS, "Data Storage System", has also evolved into "Data Sharing System" because of the software's role in sharing data between different HEC programs.

HEC-DSS Version 6 end of support was July 31, 2024.

Future Direction - It is Time for a Foundational Update

When HEC-DSS was developed in 1978, the original requirements could only be met by building HEC-DSS from the ground up, in Fortran. At that time there were no other off-the-shelf products with the flexibility HEC required, except Fortran. After 45 years of computer technology improvements, there are now many technology 'building blocks' available that can effectively meet or exceed HEC's technological requirements. Therefore, it is important to explore these newer options, because the maintenance effort for HEC-DSS has become larger than the effort to upgrade the underlying HEC-DSS technology. Adding a new feature to HEC-DSS can take months. However, that same feature could be added in weeks if HEC-DSS were built on top of modern technology.

The approach to changing the foundation of HEC-DSS needs to be strategic to ensure compatibility between incremental versions, because HEC-DSS technology is itself a building block used in many HEC software products and by many developers. Our strategy considers that HEC-DSS is called by Fortran, Java, C#, Visual Basic, and other programming languages, and has the following components:

  • Organize and simplify the access to HEC-DSS into a new cross-language, cross-platform and stable API
  • Migrate HEC applications to the new API
  • Test different back-end technologies behind the new API
  • Select optimal backend-end technology

Progress has been made on each of the items above.  We are currently testing the new API within HEC-DSS with: Java, DotNet, and Python. In fact, the updated DotNet API is being used in the development of the next generation of HEC-RAS.  This new API uses a bridge that decouples the general use of HEC-DSS 7 software from the specific technology under the hood. With the bridge in place, we can investigate changing the file format technology with minimal, or no, changes to the programs relying on HEC-DSS.  In addition, we have begun investigating using modern technology building blocks to handle the bits and bytes of disk storage. For example, in some testing trials for ensemble data, we have demonstrated that SQLite and HDF5 are very capable back-end technologies.  

In addition to the components listed above, the development of the next generation of HEC-DSS technology includes the following key requirements: 

  • Efficient storage and retrieval of both regular and irregular interval time-series data that balances read/write speeds with compression
    • Ability to efficiently read a time-window within a time-series
    • Ability to update values within a time-series
    • Ability to append values to an existing time-series
  • Storage and retrieval of tabular data
  • Ability to store and retrieve custom data structures
  • Compatible with modern programming languages
  • Support metadata such as tags/values for each type of data stored

HEC is excited about improving the foundation of the HEC-DSS Data Storage System. The new foundation will simplify adopting and testing new storage technology, while keeping an API layer between the read/write disk technology and applications of the HEC-DSS library. Consequently, this API layer allows us to be ready for the many future technological innovations that are surely coming.

Do you have an idea to Share?

Do you know of technology that could help the future of HEC-DSS?  If you have an idea to share please contact us.