Download file for R Shiny application here: uncertainty_sampling_distributions.R

Note: An Excel-based version of this workshop is available here: Task 2 - Sampling distributions for statistics of peak flow frequency curves.

Objectives

The objective of this workshop is to explore the relationship between the size of a random sample and the uncertainty in estimates of a probability distribution made from that sample.

In this second task, we'll look at the uncertainty in estimating flood flow frequency curves with the Log Pearson Type III (LP3) distribution.  We'll explore an existing spreadsheet that starts with a known LP3 distribution, generates random samples of various sizes N, re-estimates the distribution from each sample, and explores the resulting errors in the parameters and quantiles.

The spreadsheet is complex, but this exercise involves just exploring it.

Steps

Step 1: Launch Application

In this section, we'll look at uncertainty in the sample estimates of the parameters and quantiles of a Log Pearson Type III peak flow frequency curve, captured by the sampling distributions. We'll be drawing random samples of size 10 and 50 from the frequency curve.

1. Launch R Studio and open the provided .R file
2. Along the top bar, press the green play button to run the R Shiny app. You may need to install packages if they are not already installed.
   
3. There are three tabs you will navigate between in this workshop: Population, Single Experiment, and Multi Experiment
   
4. General tips
   1. Feel free to resize the window as needed to see all the content
   2. The input bar (on the left) can be expanded and hidden by clicking the > and < symbol at the top
   3. Plots can be expanded to full-size by clicking the arrows icon in the bottom-right corner of each plot
   4. The Single Experiment and Multi Experiment sections of this workshop use a random number generator, which means each time you click "Run", a new random sample will be drawn

Step 2: Population Distribution

We will start on the Population tab.

1. The left bar contains the population ("true" or "known") distribution parameters for Mean, Standard Deviation, and Skew. These represent the theoretical population parameters for a Log Pearson Type III flow frequency distribution.
2. Leave the default values as 3.6 for Mean, 0.5 for Standard Deviation, and 0.3 for Skew.
3. Hit "Run" to populate the plots.
4. The top-left window contains a table of flows per exceedance probability
5. The top-right window plots the Cumulative Distribution Function (CDF) of the population curve as a flow frequency curve
6. The bottom-left window plots the Probability Density Function (PDF) of flows
7. The bottom-right window plots the Probability Density Function (PDF) of log flows
8. Familiarize yourself with the actual population frequency curve. Feel free to change the Mean, Standard Deviation, and Skew values and re-run the results to view the effect of changing these parameters on the various plots.

Note on Kp,g

Notice throughout this workshop, the B17B estimate of Kp,g for skew between 1 and -1 is used. That estimate (called the Wilson-Hilferty transformation) is calculated as:

$//<![CDATA[ \begin{array}{l}K_{p,g} = \frac{2}{g}\left(\left(\left(Z_p - \frac{g}{6}\right)*\frac{g}{6} + 1\right)^3 - 1\right)\end{array} //]]>$

where g = skew and p = exceedance probability

Step 3: Single Experiment

Navigate to the Single Experiment tab.

1. The left bar contains the population ("true" or "known") distribution parameters, along with inputs for two random samples
2. Leave the default values as 3.6 for Mean, 0.5 for Standard Deviation, and 0.3 for Skew.
3. Enter a sample size of 10 for Sample 1, and 50 for Sample 2. Hit "Run" to generate a random sample of each specified size. Note that we are sampling from the population.
4. Each time you hit "Run" a new random sample will be generated.
5. Along the top, the Mean, Standard Deviation, and Skew estimated from each sample is shown. Notice how these differ from the true (population) parameters.
6. The bottom-left plot shows the Probability Density Function (PDF) of the true (population) log flows in green. The sampled flows for Sample 1 is shown in blue, and for Sample 2 is shown in red. The mean flow from each sample is plotted with a dashed vertical line.
7. The bottom-right plot shows the true (population) LP3 frequency curve in green, and the sampled frequency curves for Sample 1 and Sample 2 in blue and red, respectively.
8. Hit "Run" several times to see several samples of size N=10 (blue) and N=50 (red).
9. Play around with changing the sample sizes and re-running the experiment.
10. Try also changing the true (population) parameters to see the effect of the estimators in each sample.

Reflect on the typical sample sizes for flow gages in the US. Imagine you are trying to estimate the population flow frequency curve from Sample 1 or Sample 2. Do you think these sample sizes are sufficiently large to have confidence in your estimates?

Step 4: Multiple Experiments

Navigate to the Multi Experiment tab.

1. This section performs the same analysis as the Single Experiment tab (drawing two size N samples from the population), however it repeats the experiment an n number of times. For example, if the sample size is 10 and the number of experiments is 500, we will draw 500 independent samples of 10 from the population, yielding 500 independent estimators of Mean, Standard Deviation, and Skew.
2. Leave the default values as 3.6 for Mean, 0.5 for Standard Deviation, and 0.3 for Skew.
3. Enter a sample size of 10 for Sample 1 and 50 for Sample 2. Enter 1,000 for the number of experiments to run. Hit "Run" to run the experiments.
4. Each time you hit "Run", new random samples will be generated.
5. At the top, the average (mean) of the estimators (Mean, Standard Deviation, and Skew) from each experiment is shown under "Average Across Experiments". Use the radio button to view "All Experiments" to view the estimators from Sample 1 and from Sample 2 for each experiment.
6. In the lower half, notice there are three sections: Sampling Distribution of Parameters, Sampling Distribution of Q, and Sampled Curves.
   1. Sampling Distribution of Parameters: This section plots histograms for the estimators (Mean, Standard Deviation, and Skew) from each of the n=1000 experiments. Sample 1 is shown in red and Sample 2 is shown in blue.
   2. Sampling Distribution of Q: In this section, you can select a flow quantile (in years) to view estimates of the n-year flow computed from the fitted LP3 distribution for each sample.
   3. Sampled Curves: This section plots the fitted LP3 flow frequency curve for each sample compared to the population curve. Note, if the number of experiments exceeds 250, only the first 250 curves will be shown.

Step 5: Questions/Critical Thinking

Sampling Distribution of an Estimate of the MEAN

Navigate to the Multi Experiment | Sampling Distribution of Parameters section.

The collection of 1000 sample estimators (Mean, Standard Deviation, and Skew) for sample size N=10 and sample size N=50 are shown. (Note, these are the statistics of the sampling distribution of an estimate of the mean, standard deviation, and skew). Recall, the true population parameters are Mean=3.6, Stdev=0.5, and Skew=0.3. A histogram is generated from the 1000 estimates of the mean from sample size N=10 (red) and N=50 (blue), showing the sampling distribution itself.

Note, the sampling distribution’s anticipated mean is the population mean of mu μ =3.6, because the estimator for the mean is unbiased.  The sampling distribution’s anticipated standard deviation is the population standard deviation divided by the square root of N, σ/sqrt(N) or 0.5/sqrt(10).

For N=10, the anticipated standard deviation of the mean is 0.16.

For N=50, the anticipated standard deviation of the mean is 0.07.

Question 1: How does the histogram for the mean of a sample of size N=10 compare to the one for size N=50?

Question 2: Is this difference explained by the anticipated sampling distribution parameters?

Sampling Distribution of an Estimate of the STANDARD DEVIATION

Navigate to the Multi Experiment | Sampling Distribution of Parameters section.

The collection of 1000 sample estimators (Mean, Standard Deviation, and Skew) for sample size N=10 and sample size N=50 are shown. (Note, these are the statistics of the sampling distribution of an estimate of the mean, standard deviation, and skew). Recall, the true population parameters are Mean=3.6, Stdev=0.5, and Skew=0.3. A histogram is generated from the 1000 estimates of the mean from sample size N=10 (red) and N=50 (blue), showing the sampling distribution itself.

 Note, now we're exploring the sampling distribution of an estimate of the sample standard deviation from sample size N, rather than an estimate of the mean. 

The sampling distribution's expected mean is again the population value, in this case the population standard deviation of sigma σ=0.5, and the sampling distribution's anticipated standard deviation is σ/sqrt(2N). (NOTE, this is the standard deviation of the standard deviation!) .

Question 3: How does the skew of the 1000 sample estimates of standard deviation from size N=10 compare to the skew of the 1000 sample estimates of standard deviation from size N=50?   What might explain the difference?

Sampling Distribution of an Estimate of the SKEWNESS COEFFICIENT

Navigate to the Multi Experiment | Sampling Distribution of Parameters section.

The collection of 1000 sample estimators (Mean, Standard Deviation, and Skew) for sample size N=10 and sample size N=50 are shown. (Note, these are the statistics of the sampling distribution of an estimate of the mean, standard deviation, and skew). Recall, the true population parameters are Mean=3.6, Stdev=0.5, and Skew=0.3. A histogram is generated from the 1000 estimates of the mean from sample size N=10 (red) and N=50 (blue), showing the sampling distribution itself.

 Note, now we're exploring the sampling distribution of an estimate of the sample skew from sample size N. 

Note that the estimator for skew is BIASED. Skew estimates from limited random samples are closer to zero than the population skew (so, for a skew of 0.3, estimates are too low) and the sampling distribution of skew estimates itself has a positive skew for positive population skews.

Question 4: Do the skews of the 1000 estimates of sample skew agree with the above discussion for N=10 and N=50?

Sampling Distribution of an Estimate of Q-100 (the 100-year or 1% chance flow)

Navigate to the Multi Experiment | Sampling Distribution of Q section.

This section displays the estimates of the n-year flow computed from the fitted Log Pearson Type III (LP3) distribution for each sample. From the drop-down, select a flow quantile of 100 years and view the plots. Note the positive skew of the 1000 estimates of Q100 for N=10 and N=50.

Question 5: Are the positive skews of the 1000 estimates of Q100 expected?

The plots for p(Q) and Zp(Q) show the fitted frequency curves’ estimates of the probability of the actual n-year flow, for sample sizes N=10 and N=50. Note, the true values are 0.01 for the 100-year flow and 0.02 for the 50-year flow. Examine how the probabilities are estimated for each sample. The sampling distributions have tremendous skew, as seen by both the computed skew (positive) and the shape of the histograms.

Uncertainty in Sampled Curves

Navigate to the Multi Experiment | Sampled Curves section.

This section plots the fitted LP3 curve for each sample (each "realization") compared to the population curve. Note that only the first 250 curves are shown if the number of experiments exceeds 250 (for easier visualization).

Question 6: Compare the confidence limits for Sample 1 (N=10) and Sample 2 (N=50). What would you expect to happen if you increase the sample size of Sample 2 to 500?

Conclusion

Continue running experiments with varying sample sizes for Sample 1 and Sample 2, varying population parameters, and varying number of experiments to further explore the effects of these on uncertainty in flow frequency curves.