HEC-HMS calculates and displays four summary statistics to quantify model performance compared to observations. Statistics include Nash-Sutcliffe Efficiency (NSE), Ratio of the Root Mean Square Error to the Standard Deviation Ratio (RSR) and Percent Bias (PBIAS) (Moriasi, et al., 2007), as well as Coefficient of Determination (R2) (Legates and McCabe, 1999). These statistics are summarized in the Table below. 

Table. Calibration summary statistics

Criterion

Equation

Notes

Nash-Sutcliffe Efficiency (NSE)


1) NSE=1-\left[\frac{\sum_{i=1}^{n}(Y_i^{obs} - Y_i^{sim})^2} {\sum_{i=1}^{n}(Y_i^{obs} - \bar Y_{obs})^2} \right]


  • NSE evaluates the relative difference between the magnitude of residual data variance ("noise") and the measured data variance (“information”)
  • It is a measure of how well the plot of observed versus simulated data fits the 1:1 line. 
  • Ranges between -\infty and 1.
  • NSE of 1 indicates one-to-one match between the observed and simulated values
  • Negative NSE indicates that the mean of observed values provides a better estimate of the observed data than the model.
  • Widely used in hydrology and is considered a good statistic to represent the overall shape of the hydrograph.

Ratio of the Root Mean Square Error to the Standard Deviation Ratio (RSR)

2) RSR=\frac{RSME}{\sigma_o}=\frac{\left[\sqrt{\sum_{i=1}^{n}(Y_i^{obs}-Y_i^{sim})^2}\right]}{\left[\sqrt{\sum_{i=1}^{n}(Y_i^{obs}-\bar Y_{obs})^2}\right]}
  • Standardizes the root mean square error (RMSE) using the observation standard deviation
  • The optimal value of RSR is 0.0 which presents zero RMSE for perfect model simulation
  • Lower RSR values present better model performance

Percent Bias (PBIAS)

3) PBIAS=\left [\frac{\sum_i^n(Y_i^{obs}-Y_i^{sim})*(100)} {\sum_i^n(Y_i^{obs})}\right]
  • Provides a measure of the simulated values are on average larger or smaller than the corresponding observed values.
  • Varies from 0% to infinity, with 0 being the optimal. 
  • Positive PBIAS means the model under-estimates observed data
  • Negative PBIAS means the model over-estimates observed data

Coefficient of Determination (R2)


4) R^2=\left[\frac{\sum_i^n(Y_i^{obs}-\bar Y_{obs})(Y_i^{sim}-\bar Y_{sim})}{\sqrt{\sum_i^n(Y_i^{obs}-\bar Y_{obs})^2}*\sqrt{\sum_i^n(Y_i^{sim}-\bar Y_{sim})^2}}\right]^2
  • R2 describes degree of collinearity between simulated and observed data.
  • Represents the proportion of the variance in measured data explained by the model. 
  • Varies between 0 and 1, with 1 being an optimal value.
  • Is oversensitive to outliers and insensitive to additive and proportional differences between model predictions and measured data

Modified Kling Gupta Efficiency (MKGE)

5) MKGE = 1 - \sqrt{(r - 1)^2 + (\beta - 1)^2 + (\gamma - 1)^2}
6) \beta = \frac{\bar Y_{sim}}{\bar Y_{obs}}
7) \gamma= \frac{CV_s}{CV_o} = \frac{\sigma_s/\bar Y_{sim}}{\sigma_o/\bar Y_{obs}}


  • Multi-objective alternative to mean squared error and Nash-Sutcliffe Efficiency (NSE)
  • Can be decomposed into three terms: (1) correlation r, (2) bias ratio \beta, and (3) variability ratio \gamma
  • The value of MKGE gives the lower limit of the three components (r, \beta\gamma)
  • The original version of the KGE-statistic uses a variability ratio of γ of σs/σo instead of CVs/CVo. The ratio of coefficient of variations, rather than standard deviations, ensures that the bias and variability ratios are not cross-correlated.


Variables : 

  • Y_i^{obs} = ith observation 
  • Y_i^{sim} =  ith simulated value
  • \bar Y_{obs} = the mean of observed data
  • \bar Y_{sim} = the mean of simulated data
  •  n = total number of observations
  • r = correlation coefficient between simulated and observed runoff (dimensionless)
  • \beta = bias ratio (dimensionless)
  • \gamma = variability ratio (dimensionless)
  • CV = coefficient of variation (dimensionless)
  • \sigma = standard deviation
  • The indices s and o represent simulated and observed runoff values, respectively.

HEC-HMS also reports observed and computed maximum flow, time of peak and total volume. These measures are also useful in the calibration process.

As a reminder, the following basic statistical measures are useful for this discussion:

  • Residual variance  = sum of squared differences between the observed and simulated values = \sum_{i=1}^{n}(Y_i^{obs} - Y_i^{sim})^2
  • Measured data variance = sum of squared differences between the individual observed values and the mean of the observed value = \sum_{i=1}^{n}(Y_i^{obs} - Y_{obs}^{mean})^2
  • Standard deviation (\sigma) is the square root of variance

Performance ranges

Suggested model performance ranges of the four summary statistics for evaluating streamflow, adapted from Moriasi et all, 2007 and 2015, are summarized in the Table below. Note that these are derived for continuous flow data at daily and monthly time step at watershed scale. The acceptable values of the summary statistics will vary for your project, depending on the time step, uncertainty in observed data and boundary conditions and project scope. 

Table. HEC-HMS Performance Ratings for Summary Statistics

Performance Rating

NSE

RSR

PBIAS (%)

R2

Very Good

0.75<𝑁𝑆𝐸≤1.00

0.00<𝑅𝑆𝑅≤0.50

|𝑃𝐵𝐼𝐴𝑆| < ±10

R2 ≥ 0.85

Good

0.65<𝑁𝑆𝐸≤0.75

0.50<𝑅𝑆𝑅≤0.60

±10≤ |𝑃𝐵𝐼𝐴𝑆| <±15

0.70≤R2<0.85

Satisfactory

0.50<𝑁𝑆𝐸≤0.65

0.60<𝑅𝑆𝑅≤0.70

±15≤ |𝑃𝐵𝐼𝐴𝑆| <±25

0.5≤R2<0.70

Unsatisfactory

𝑁𝑆𝐸≤0.50

𝑅𝑆𝑅>0.70

|𝑃𝐵𝐼𝐴𝑆| ≥±25

R2≤0.5

Performance ranges for MKGE are not provided in HEC-HMS. Using the mean flow as a predictor results in a NSE = 0 and a MKGE = 1 - \sqrt2 = -0.41. MKGE values greater than -0.41 indicate that the model's performance is better than the mean flow. NSE and MKGE values cannot be directly compared because the relationship depends in part on the coefficient of variation of the observed time series (Knoben et al., 2019). Modelers should analyze the MKGE components (correlation coefficient, bias ratio, and variability ratio) to better understand the model error.