Download PDF
Download page Validation Guide.
Validation Guide
Introduction
The Statistical Software Package (HEC-SSP) is designed to perform statistical analyses of hydrologic data. Both time series (e.g. annual maximum series of flow) and paired data can be analyzed using a wide array of analysis options including flow- and volume-frequency, duration, coincident frequency, balanced hydrograph creation, distribution fitting, mixed population, correlation, and streamflow record extension, amongst others.
The interaction of these various data and simulation components leads to a software package that is very complex. The U.S. Army Corps of Engineers (USACE) support planning, design, operation, permitting, and regulating activities by providing information about current and potential flow-, precipitation-, and stage-frequency, amongst others. HEC-SSP can support these activities by quantifying statistical relationships. It is critical that these estimates be precise and accurate if they are to be useful. Precision implies that every time the software computes a simulation, it will obtain the exact same numerical results for the same input data. Accuracy implies that the foundational equations of each analysis are solved correctly.
Code validation can be described as the process of determining that a complex software program such as HEC-SSP produces precise and accurate results. Code validation should not be confused with other types of testing. For example, model validation is the process of determining if a proposed model (often a mathematical model) accurately represents a physical process and provides predictive capability. Code validation focuses on the very specific question of whether the software accurately solves the equations used to model or represent the physical process. Software should not be used for any of the tasks undertaken by a water resources engineer unless it first can demonstrate code validation of all its internal components.
The models and equations incorporated in HEC-SSP have been validated by the research community. The equations in each of the simulation components included in the program have been sourced from published statistical hydrology textbooks and from papers published in peer-reviewed journals. The code validation process described here is designed to determine if the published models and equations have been incorporated correctly in the HEC-SSP software. However, code validation does not mean that a particular analysis option can be applied in any and every watershed. A variety of analysis capabilities are included in HEC-SSP because every application is unique. Users are responsible for selecting simulation components, configuring them to appropriately represent the relevant processes in the watershed, and verifying their results are appropriate.
The Test Suite
The team responsible for the development of HEC-SSP has created an extensive set of code validation tests. The tests are designed to determine if input data, including time-series and paired data, are processed correctly. This includes tests of unit system conversion and interpolation. The code validation tests also determine if the analysis results are accurate. The accuracy of the results is determined by one of two methods. In some cases it is possible, given the input data and parameters, to calculate the correct result by hand. In these cases the equations are simple enough that hand calculations with the aid of a calculator can determine that the results computed by HEC-SSP are correct. Other cases require independent software such as Microsoft Excel or the R Statistical Language. This second approach is required when the equations used in an analysis component are differential equations or some other type of complex equation. Excel and the R Statistical Language are developed independently by teams of industry specialists. The results from these independent software tools can be used to confirm that HEC-SSP is calculating accurate results.
The code validation test suite created for HEC-SSP includes numerous projects. Each of these projects contains many analyses. Each analysis tests one or more components of the software. These analyses were created because they are complex and require the interaction of many different options included in HEC-SSP. These tests have been carefully developed and have a known or desired solution. Aside from testing for accuracy and precision, these analyses are used to detect any changes in simulated values that may arise from incremental changes in the software. A change in the software that causes a computed result to change can be detected with these projects. If such a change occurs, then the source of the change is found, investigated, and resolved.
Five separate analyses are used within the validation test suite. The names of these projects are:
- SSP EXAMPLES
- Bulletin 17C Examples
- Additional Tests
- 82 Test Sites
- Paleoflood Tests
Tables detailing these projects and analyses are shown below:
More information pertaining to these projects can be found here: HEC-SSP Examples.
SSP EXAMPLES Project Used for Testing HEC-SSP
| Analysis Name | Analysis Type | Option |
|---|---|---|
| FFA Test 1 | Bulletin 17 | Bulletin 17B |
| FFA Test 2 | Bulletin 17 | Bulletin 17B - High Outliers |
| FFA Test 3 | Bulletin 17 | Bulletin 17B - Low Outliers |
| FFA Test 4 | Bulletin 17 | Bulletin 17B - Zero Flood Years |
| FFA Test 5 | Bulletin 17 | Bulletin 17B - Confidence Limits and Low Threshold Discharge |
| FFA Test 6 | Bulletin 17 | Bulletin 17B - Historical Data |
| Sinnemahoning_B17C | Bulletin 17 | Bulletin 17C |
| KettleCreek_Summer_B17C | Bulletin 17 | Bulletin 17C |
| KettleCreek_Winter_B17C | Bulletin 17 | Bulletin 17C |
| Bushkill_MovingWindow | Bulletin 17 | Bulletin 17C - Moving Window |
| Bushkill_ExpandingWindow | Bulletin 17 | Bulletin 17C - Expanding Window |
| General Frequency FFA Test 1 | General Frequency | Bulletin 17B |
| General Frequency FFA Test 2 | General Frequency | Bulletin 17B - High Outliers |
| General Frequency FFA Test 3 | General Frequency | Bulletin 17B - Low Outliers |
| General Frequency FFA Test 4 | General Frequency | Bulletin 17B - Zero Flood Years |
| General Frequency FFA Test 5 | General Frequency | Bulletin 17B - Confidence Limits and Low Threshold Discharge |
| General Frequency FFA Test 6 | General Frequency | Bulletin 17B - Historical Data |
| Stage Analysis Test 7 | General Frequency | Graphical Analysis |
| Local Runoff at Alb Test 8 | General Frequency | Bulletin 17B |
| Reg Flow Rio Grande Test 9 | General Frequency | Graphical Analysis |
| General Frequency EMA Test | General Frequency | Bulletin 17C |
| General Frequency Precip Test | General Frequency | Precipitation Frequency |
| Unreg Flow Rio Grande Test 10 | Volume Frequency | Volume Frequency using B17B Procedures |
| Low Flow Analysis Test 11 | Volume Frequency | Low-Flow Volume Frequency using B17B Procedures |
| Sinnemahoning_VolumeFrequency | Volume Frequency | Volume Frequency using B17C Procedures |
| Fishkill Creek Test 12 | Duration | Duration using STATS Method |
| Duration Curve Test 13 | Duration | Duration using All Data Values Method |
| Manual Duration Curve Test 14 | Duration | Duration using Manual Method |
| Coincident Freq Test 15 | Coincident Frequency | A and B can be Assumed Independent |
| Coincident Freq Test 16 | Coincident Frequency | A and B can not be Assumed Independent |
| Curve Combination Test 19 | Curve Combination | Combining Multiple Frequency Curves Using Graphical Techniques |
| Balanced Hydrograph Test 17 | Balanced Hydrograph | Using an Historical Event Hydrograph |
| Balanced Hydrograph Test 18 | Balanced Hydrograph | Using a Triangular-shaped Hydrograph |
| Distribution Fitting Test 20 | Distribution Fitting | Analyzing Model Uncertainty Using a Time Series of Annual Maximum Streamflow |
| Distribution Fitting Test 21 | Distribution Fitting | Analyzing a Time Series of Daily Average Flow Using Time Window, Seasonal, and Annual Maxima Filtering |
| Distribution Fitting Test 22 | Distribution Fitting | Analyzing a Time Series of Daily Average Flow Using Time Window, Seasonal, and Peaks Over Threshold Filtering |
| Distribution Fitting Test 23 | Distribution Fitting | Analyzing a Time Series of Daily Precipitation Accumulation Using Peaks Over Threshold Filtering |
| Distribution Fitting Test 24 | Distribution Fitting | Analyzing a Time Series of Daily Average Snow Water Equivalent Using Annual Maxima Filtering |
| Distribution Fitting Test 25 | Distribution Fitting | Analyzing a Time Series of Daily Average Flow to Estimate Flood Seasonality |
| Distribution Fitting Test 26 | Distribution Fitting | Analyzing a Paired Data Record of Hydrologic Model Output |
| Mixed Population Test 27 | Mixed Population | Combining Two Frequency Curves |
| Mixed Population Test 28 | Mixed Population | Combining Three Frequency Curves |
| WBSusquehanna_Correlation | Correlation | Correlation for Four Annual Maximum Peak Discharge Time Series |
| TarRiver_Correlation | Correlation | Correlation for an Annual Maximum Peak Discharge and a Daily Average Tidal Stage Time Series |
| PairedData_Correlation | Correlation | Correlation for Four Paired Data Curves |
| MFEelRiver_RecordExtension | Record Extension | MOVE.3 (Bulletin 17C) Record Extension for Annual Maximum Peak Discharge Time Series |
| YellowstoneRiver_RecordExtension | Record Extension | MOVE.1 Record Extension for Daily Average Flow Time Series |
Bulletin 17C Examples Project Analyses Used for Testing HEC-SSP
| Analysis Name | Analysis Type | Option |
|---|---|---|
| Bulletin 17C Example 1 | Bulletin 17 | Bulletin 17C - Systematic Record |
| Bulletin 17C Example 2 | Bulletin 17 | Bulletin 17C - Analysis with Low Outliers |
| Bulletin 17C Example 3 | Bulletin 17 | Bulletin 17C - Broken Record |
| Bulletin 17C Example 4 | Bulletin 17 | Bulletin 17C - Historical Data |
| Bulletin 17C Example 5 | Bulletin 17 | Bulletin 17C - Crest Stage Gage Censored Data |
| Bulletin 17C Example 6 | Bulletin 17 | Bulletin 17C - Historic Data and Low Outliers |
| Bulletin 17C Example 7 | Bulletin 17 | Bulletin 17C - User Overrides (Low Outliers and Confidence Limits) |
| Bulletin 17C Example 8 | Bulletin 17 | Bulletin 17C - [inf – inf] Perception Threshold |
| Bulletin 17C Example 9 | Bulletin 17 | Bulletin 17C - [0 – value] Perception Threshold |
| Bulletin 17C Example 10 | Bulletin 17 | Bulletin 17C - Systematic and Historical Record with Regional Skew |
Additional Tests Project Analyses Used for Testing HEC-SSP
| Analysis Name | Analysis Type | Option |
|---|---|---|
| Extra_at_the_Beginning | Bulletin 17 | Bulletin 17C - Manually Entered Data at the Beginning of the Time Series |
| Extra_at_the_End | Bulletin 17 | Bulletin 17C - Manually Entered Data at the End of the Time Series |
| CENS_in_the_middle | Bulletin 17 | Bulletin 17C - Manually Censored Data |
| Extra_on_both_sides | Bulletin 17 | Bulletin 17C - Manually Entered Data at the Beginning and End of the Time Series |
| SantaCruz_B17B | Bulletin 17 | Bulletin 17B - Historic Data and Low Outliers |
| Trenton_MovingWindow | Bulletin 17 | Bulletin 17C - Moving Window |
| Trenton_ExpandingWindow | Bulletin 17 | Bulletin 17C - Expanding Window |
| Trenton_B17C | Bulletin 17 | Bulletin 17C |
| Very_Small_ACE | General Frequency | Bulletin 17C - Very Small Probability Output Ordinates |
| Orofino_GPA | General Frequency | Flow Frequency Using Generalized Pareto Distribution |
| StateLine_GEV | General Frequency | Flow Frequency Using Generalized Extreme Values Distribution |
| SantaCruz_B17B_MGB | General Frequency | Bulletin 17B - Historic Data and Low Outliers With Multiple Grubbs-Beck Test |
| Orestimba_B17B_MGB | General Frequency | Bulletin 17B - Historic Data and Low Outliers With Multiple Grubbs-Beck Test |
| CalendarYear_EMA | General Frequency | Bulletin 17C - Using Calendar Year Option |
| CalendarYear_B17B | General Frequency | Bulletin 17B - Using Calendar Year Option |
| OtherYear_EMA | General Frequency | Bulletin 17C - Using Other Year Option |
| OtherYear_B17B | General Frequency | Bulletin 17B - Using Other Year Option |
| Orestimba_Maxima | Volume Frequency | Volume Frequency - B17B Procedures, Analyze Maxima, Using Time Window Modification |
| Orestimba_Maxima_EMA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Using Time Window Modification |
| Chemung_Minima | Volume Frequency | Volume Frequency - B17B Procedures, Analyze Minima, Using Time Window Modification |
| Chemung_Minima_EMA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Minima, Using Time Window Modification |
| B17B_Exp_Prob | Volume Frequency | Volume Frequency - B17B Procedures, Analyze Maxima, Using Time Window Modification, Compute Expected Probability |
| B17C_Exp_Prob | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Using Time Window Modification, Compute Expected Probability |
Keystone_Unreg_VFA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Custom Durations, Manually Entered Data |
| RedRiver_VFA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Other Year Option, Using Time Window Modification |
| TheDalles_VFA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Using Time Window Modification, Seasonal Subset of the Year |
| Aquashicola_EMA | Volume Frequency | Volume Frequency - B17C Procedures, Analyze Maxima, Other Year Option, Using Time Window Modification |
| ExtraOnBothSides_VDF | Volume Frequency | Volume Frequency - B17C Using Manually Entered Data at the Beginning and End of the Time Series |
| LessThanFullRecord_VDF | Volume Frequency | Volume Frequency - B17C Using Less Data Than Contained in the Flow Data Set |
82 Test Sites Project Analyses Used for Testing HEC-SSP
| Analysis Name | Analysis Type | Option |
|---|---|---|
| <all analyses>* | Bulletin 17 | Bulletin 17C |
*these tests represent all sites evaluated within Evaluation of Recommended Revisions to Bulletin 17B (Cohn, et at,. 2017)
Paleoflood Tests Project Analyses Used for Testing HEC-SSP
| Analysis Name | Analysis Type | Option |
|---|---|---|
| <all analyses> | Bulletin 17 | Bulletin 17C - Paleoflood Data, Non-Exceedance Bounds, Paleo-stage Indicators, Regional Skew, and DSS Extended Dates |
The projects used in the code validation test suite compute hundreds of time-series and paired data records. Each value of the time-series or paired data record is compared and a test is deemed to have failed if even a single value is outside the allowable range. As a general rule, tests use an allowable error range of 0.02% of the correct value. This means that HEC-SSP is deemed to have failed a validation test if it makes an error greater than 0.02% of the correct value. The development team reviews the results from the entire test suite before certifying a new software version for general release. The validation suite is used to find errors so they can be repaired.
A new software version is not released for general use until it can successfully pass all of the code validation tests in the suite. We make the test suite available through the HEC website (www.hec.usace.army.mil) so that users may, if desired, review the data sets included in the suite, and optionally compute them and compare the results to the benchmark values. The projects do not need to be evaluated after downloading the software to know that the software is working correctly. The software is fully tested before it is released for use and therefore will pass all of the tests successfully after it is installed. The digital certificate included in the installation package guarantees that the software has not changed since it was certified as passing all validation tests. Any change that might occur in the software between the time it leaves HEC and the time it is actually installed could cause errors during validation. Because a digital certificate is included in the installation package, the user would be alerted to any such changes during the installation process. Nevertheless, users have the option of independently confirming the results already achieved by HEC staff preparing the software for release.
The following sections serve as documentation of the procedures used by HEC staff during testing. All of these steps are followed by HEC staff during the testing required before a new version of HEC-SSP is released for use. A user could follow these same steps if they wished to independently validate the software.
Any changes to the data in the validation projects will cause the tests to fail.
Established Accurate Results
Hand calculations and independent software have been used to establish known accurate results for each code validation test. These known results are stored in a DSS file. There is a separate DSS file for each of the corresponding projects used in validation testing. The simulation results in each DSS file have been checked very carefully. These files should never be modified.
Automated Comparison
It would be very onerous to compare all of the time-series and paired data records manually. The development team has created a small utility program to automate this task. The utility program reads the time-series and paired data of computed results from the project DSS file. It also reads the records from the known result. The utility program then compares value by value the two records. If two values do not agree within the error range, the error is recorded in a log file. A successful passage of the test is recorded in the log file if all values match within the necessary error range. The log file is the only place to determine the success or failure of each test.