Last Modified: 2026-04-27 06:55:56.665

HEC-HMS version 4.14 beta 1 was used to created this tutorial. You will need to use HEC-HMS version 14 or newer, to open the project files.

Download the initial model files here - CrossValidation_Initial.zip

Background

To perform a cross-validation, each cross-validation run must have an associated basin model that uses validation parameters from a given event and initial conditions from validation period. Previously this required the creation of serveral basin models. For example, if we wanted to know how a set of parameters calibrated to a December 1996 event perform when simulating three other events: December 2015, March 2019 and February 2022, we would need to create three basin models. Each basin model would use a given set of initial conditions (initial loss parameter, initial groundwater, initial canopy storage etc.) and December 1996 parameter values for all other model parameters. Furthermore, if we wanted to repeat the process for the remaining three events, it would result in 12 basin models. Cross-validation allows the user to see how a given set of calibrated parameters perform over different validation periods with different initial watershed conditions. This helps to inform which parameter values are best suited to represent the watershed under diverse conditions. An automated cross-validation process within HMS streamlines this workflow and helps the user more efficiently select parameters and highlight potential outliers.

Objectives

• Set up a Cross-Validation Run
• Run a Cross-Validation Run
• Interpret and Verify Results

Review the Model

The existing model has three simulation runs with varying parameters, meteorologic methods, and control specifications. One essential shared characteristic across all basin models is that they all have the same element names, methods, and element types. If any basin model has extra or missing elements, the Cross-Validation Analysis will error and fail to run. The underlying code relies on swapping initial conditions and parameter values across basin elements that are identical in all ways besides the parameter value itself.

All basin element names and methods must be the same across all members. Meteorologic Methods do not need to be identical. Observed data used for calibration will use whatever gage is referenced in the initial condition model, so these can also vary.

Create a New Cross-Validation Analysis

1. Navigate to the Compute menu on the toolbar. Select Create Compute | Cross-Validation.
2. Name your new analysis Punx Cross-Validation. Keep the Analysis Type set to Simulation Run and click Next.
3. Select all three Simulations Runs, click Next and then Finish.

View the Cross-Validation Members

Navigate to the compute tab, expand the Cross-Validation folder.  Select the Punx Cross-Validation that you just created and look at the Cross-Validation Members tab. Notice the Simulation Run Name, Basin Model and Meteorologic model correspond to the existing Simulation Runs. Those associated Basin Models and Meteorologic Models are greyed out and cannot be changed. Simulation Runs can be added or removed from a Cross-Validation Run analysis from this editor or the Cross-Validation Manager in the Compute menu from the toolbar. 

Run the Cross-Validation Analysis

From the toolbar, select Punx Cross-Validation from the compute dropdown and click the  icon to run your Cross-Validation analysis. Alternatively, you can compute the run from the Compute Tree or the Compute menu on the toolbar. The dialog will cycle through six total runs, one for each Cross-Validation Member Run. Each Cross-Validation Member Run is a combination of two of the Simulation Runs selected in your Cross-Validation, it combines the forcing and initial conditions of one simulation run and the basin model parameters of another simulation run.

View Calibration Summaries

Navigate to the Results tab and expand the Cross-Validation node. Notice there are five summary tables representing the five calibration metrics available in HEC-HMS: Nash-Sutcliffe (NSE), Root Mean Squared Error (RMSE). R2, Percent Bias (PBIAS) and Modified Kling-Gupta Efficiency (MKGE). Within these tables you can view how each Cross-Validation Member Run performs at each of the designated Computation Points against the observed data.

Naming Conventions

The naming conventions of a Cross-Validation Member Run is made up of the name of the Initial Condition or Forcing Model and the Validation Model which supplies all parameters besides those designated as initial conditions. In this example, a Cross-Validation Member Run named April 1996- Sep 2018 uses a Meteorologic Model, Control Specification, Observed Data, and Initial Conditions from the April 1996 event. All other parameters use the calibrated values from the Sep 2018 Simulation Run.

Open the NSE table and examine the values. It appears the April 1996 parameters have the best NSE values across different forcing conditions. To more easily view metrics by Basin Model parameter sets, filter with the Basin Model dropdown. The listed Basin Model represents the Validation Model from which the majority of parameter values are pulled.

View Member Run Results

Under the calibration summary tables are folders titled Forcing: <Basin Model Name>. This organizes results by the forcing conditions used in the run, for example, Forcing: Apr1994 folder contains all the member runs that use the April 1994 simulation's initial conditions, meteorologic model, and control specifications. Within each member run name, the second basin model after the dash) will indicate the basin model used for all parameters (aside from initial conditions). In the results tree shown below, the element results listed under the Apr1994-Apr1996 node are no different than a normal Simulation Run and can be viewed and interpreted in a similar fashion.

Verify Initial Conditions in Cross-Validation

Let's take a look to prove that the initial conditions were appropriately swapped within the member runs. Let's compare the Apr1994 and Apr1996 basin models. Most initial conditions are hard to visualize on a plot, however initial baseflow is fairly straightforward to visually interpret. The Apr1994 basin model uses an initial baseflow in layer 1 of 2 cfs/ sq. mi and the Apr1996 basin model uses an initial baseflow of 1 cfs/ sq. mi for the East Branch Mahoning Creek subbasin, which is approximately 40 square miles of drainage area. We would expect for any member run with Apr1994 as the initial condition or forcing model to have an initial baseflow of about 80 cfs and similarly any member run with Apr1996 as the initial condition or forcing model to have an initial baseflow of about 40 cfs.

Let's look at the Baseflow Layer 1 results for the Apr1994-Sep2018 member run and the Apr1996-Apr1994 member run. The Apr1994-Sep2018 member run has an initial baseflow in layer 1 of about 82 cfs and the Apr1996-Apr1994 member run has an initial baseflow of 41 cfs.

Download the final model files here - CrossValidation_Final.zip