Many data sets contain erroneous values, which must be corrected before performing mathematical operations with the data.  HEC-DSSVue provides some limited screening criteria, allowing values to be declared as 'missing', and/or quality flags to be set.  This workshop demonstrates three of the four mechanisms available to identify and replace invalid data:

  • Screen Using Minimum and Maximum
  • Screen with Forward Moving Average (not used in this workshop)
  • Manual editing of values inside Tabulation window
  • Estimate Missing Values

Task 1. Select and View Data

  1. Start HEC-DSSVue and open “TimeSeriesMath.dss” 
  2. Click the Set Time Window button and select No Time Window if it is not already selected. Close the Set Time Window dialog.
  3. Select View → Condensed Catalog from the menu bar.  In the absence of a time window, this makes subsequent operations on a pathname apply to all the records, regardless of the D-part.
  4. Select View → Unit System → As Stored from the menu bar, to avoid inadvertent unit conversions when writing Math Functions results back to the database.
  5. Select the data set named
    /FOX RIVER/LUTZ PARK/FLOW-RES OUT/22Sep2000 - 31Oct2000/15Minute/USGS-CST/
  6. Plot and tabulate the data.
    1. Notice the plot indicates the shape of the good data but has lots of bad data points.
    2. Notice the table shows that the data is offset three minutes into every 15-minute interval..

    3. These data were produced by a very old USGS Acoustic Velocity Meter.  Each minute it made an instantaneous measurement of mean velocity across the channel, which it multiplied by cross-sectional area to calculate flow.  Every 15 minutes it applied some very coarse quality-control procedures, averaged the samples, and reported the resulting flow to the USGS. 

Task 2. Screen Data

In situations where the actual data changes gradually, individual fluky values can be easily screened out using a “moving average” technique.  However, this gage is located a short distance downstream of the Lake Winnebago outlet structures and hydropower facilities.  Abrupt flow changes typically indicate gage malfunction but can be difficult to distinguish from actual variations due to gate movements or generating adjustments.

  1. Launch the Math Functions dialog from the tool bar or menu bar.
  2. Choose the General tab and Screen with Forward Moving Average operator.
  3. Your engineering judgment and experience with this gage suggest that the flows cannot realistically change more than 500 cfs in an hour (four 15-minute periods).
    1. Screen the data set using the parameters shown in the figure below. (Be sure to click the Compute button)
  4. Plot and tabulate your computed data to review the changes. 
    1. Try selecting 'Original Data with Computed' under the Display menu and re-plotting the data. 
  5. Question 1. If the original curve covers up the “modified” curve in the plot, how can you switch order so that the revised curve is plotted on top of the original?
    • Go to the "Edit" button on the ribbon and select "Plot Properties..."
    • Use the arrows to the right to change the order of the curves
    • Alternatively, you can click on the desired data set in the legend to highlight it.
  6. Select View → Quality → as symbol and View → Quality → as hex from the table menu bar.
    1. Note the three different quality flags associated with the “modified” data? 
      1. 0000 0003 (hex) or empty cell (symbol) means okay value,
      2. 0000 0005 (hex) or M (symbol) means missing before the test.
      3. 0000 0011 (hex) or R (symbol) means rejected by the test.
  7. Look again at the plot and note that many dubious values persist. 
    1. The revised data is still unfit for use in computations. 
    2.  Perhaps a more aggressive screening test is needed. 
    3. Leave the existing plot open for subsequent comparisons
  8. Re-screen the data using a change value of 200 cfs over four 15-minute periods.
    1. Select Edit → Restore Original Data from Math Functions menu bar and verify that when you plot the data that only the original pathname appears. 
    2. Modify the Change Value Limit value to 200 and click the Compute button.
    3. Review the results by plotting and tabulating.
  9. Question 2. Did the stricter test reject any additional data (bad or good)?
    • Yes. You can tell by comparing the 500 cfs screening (left) with the 200 cfs screening (right)  that the bad point around the Oct 9 is no longer included in the screened data, but some good values in the steep transitions were also rejected.

  10. Since we had difficulty effectively screening the data using Screen with Forward Moving Average, let's try a different method.
    1. Restore the original data again (Edit → Restore Original Data) and verify that when you plot the data that only the original pathname appears. 
    2. Screen the data using the Screen using Minimum and Maximum operator with the values shown in the figure below.
  11. Plot and tabulate your computed data to review the changes. 
  12. Question 3. Does the revised data look ready to for use in a computation? Why or why not?

    Not ready yet because calculations (and models) generally require all values to be present.   The missing values will need to be addressed in some way.

    Also note the upward-spike on 9Oct2000, and a block of dubious data on 25Oct2000 still remain after the initial screening,.

  13. Save the results so far using File → Save As from the Math Functions dialog menu bar or the Save As icon on the Math Functions dialog tool bar.
    1. Save with the F part name of USGS-CST-REV
    2. Close the Math Functions dialog.
  14. Manually clear bad values on October 9.
    1. From the HEC-DSSVue window, clear any previously selected pathnames, and tabulate your revised data. 
    2. From the table menu bar:
      1. Select Edit → Allow Editing
      2. Make sure View → Date with 4 Digit Years is selected
      3. Make sure View → Date and Time Separately is not selected
    3. Scroll down to the values for 09 Oct 2000, 07:03 (you can also use Edit → Find... or F3 to search for the date/time)
    4. Highlight the rows of 09 Oct 2000, 07:03 and 09 Oct 2000, 07:18.
    5. Right-click on the selected values and select Clear.
    6. Save your changes to the same pathname (File → Save or Ctrl-S) and close the table.
  15. Use focused time window to screen out bad values on Oct 25.
    1. Click the Select Time Window button on the HEC-DSSVue window and set the time window as shown:
    2. Plot and tabulate the data to show that flows below 4500 cfs in this time window are likely bad
    3. Launch the Math Functions dialog using the recently created/revised data set. 
      1. Since the time window is set, the operations are confined to data for 25Oct2000. 
    4. Execute the Screen Using Minimum and Maximum with the Minimum Value Limit of 4500.
      1. You can leave the other value fields blank so that no maximum value or change limit is imposed. 
    5. Plot the modified and original data to verify the results of the operation.
    6. Save the data back to the same name and close the Math Functions dialog.
  16. Verify the results so far.
    1. Clear the time window in HEC-DSSVue, and plot the data. 
    2. All suspicious values should be gone. 
    3. Close your plot.

Task 3. Fill Gaps in Data

  1. Lauch the Math Functions dialog with the revised Lutz Park flows.
  2. Select the Estimate Missing Values operator on the General tab.
  3. Execute the computation with value of 96 to prevent estimating more than a day's worth of 15-minute values
  4. Plot and tabulate the new data set, verifying that all of the missing values have been filled in.
  5. Save your data.
  6. Close the math window if it is still open.
  7. Your data technician passes by and points out that cleaned up data already exists in /FOX RIVER/LUTZ PARK/FLOW-RES OUT//15Minute/USGS-CST-FIXED/.
    1. He thinks it’s great that you walked a mile in his shoes, but really, you should just use the “fixed” version provided by an experienced professional, pointing out that:
      1. The values on Oct 25 were valid and should not have been screened out
      2. That the parameter type must be "INST-VAL" according to an arbitrary but long-standing convention.
  8. Plot and tabulate your "-REV" data vs the "-FIXED" data.   
  9. Question 4. Does re-defining the parameter type to "INST-VAL" actually change anything?

Yes, the meaning of the timestamps has changed. What was previously the end of an averaging period is now the time of an instantaneous observation.