Download file for R Shiny application here: uncertainty_dice_roll.R

Note: An Excel-based version of this workshop is available here: Task 1 - Sampling distributions for the probability of rolling a 4 on a 6-sided die

Objectives

The objective of this workshop is to explore the relationship between the size of a random sample and the uncertainty in estimates of probability made from that sample.

In this first task, we'll look at the uncertainty in estimating the likelihood of each side of a 6-sided die resulting from a roll, as captured in confidence intervals. We'll explore an existing spreadsheet that generates random samples of die rolls (of fair dice!!) for various sample sizes, estimates probability of a roll showing each side, and compares to the known probability of 1/6. 

Consider why I say that probability is known in these experiments!

Step 1: Launch Application

In this section, we'll look at uncertainty in the sample estimates of the probability of rolling a 4 on a standard six-sided dice. We'll be rolling the die an N number of times and estimating the uncertainty from an n number of experiments.

  1. Launch R Studio and open the provided .R file
  2. Along the top bar, press the green play button to run the R Shiny app. You may need to install packages if they are not already installed.
  3. There are three tabs you will navigate between in this workshop: Single, Multiple, and Multiple Comparison
  4. General tips
    1. Feel free to resize the window as needed to see all the content
    2. The input bar (on the left) can be expanded and hidden by clicking the > and < symbol at the top
    3. Plots can be expanded to full-size by clicking the arrows icon in the bottom-right corner of each plot
    4. This workshop use a random number generator, which means each time you click "Roll", a new random sample will be drawn

Step 2: Single Experiment

Navigate to the Single tab.

This should look similar to the dice experiment on Monday morning, with information about confidence intervals added. The Single Experiment tab allows you to enter a number of rolls and view the roll outcomes, including the estimated probability (relative frequency) of each outcome (1 through 6), calculated as # outcomes / # rolls.

12 rolls. Enter a number of rolls, N, of 12 and click "Roll". The plot accessible in the top-right panel (Summary of All Rolls) shows the outcomes of each roll as well as the 90% confidence interval. These confidence intervals are calculated based on the Normal Distribution. For the Normal Distribution to be an adequate representation of the uncertainty, N*p > 5 must be true, where N = sample size and p = probability of success (probability of rolling a 4). The equations for calculating the confidence intervals are shown in the bottom-right panel.

Question 1: Is the requirement met?

No, for N=12, N*p = 2.  Therefore, the Normal approximation is not adequate here.

Question 2: Look at the lower and upper edge of the 90% confidence interval estimated from the Normal Distribution for sample size N=12. What problems do you see?

The lower edge of the 90% interval is negative (outside the bounds of the plot), which is not a possible probability.  The Normal distribution does not hold true because, with this sample size, it is possible to have an estimate with more positive error than negative error.  This asymmetry of error results because, while 2 fours is the expected value (mean) from 12 rolls, many more than 2 fours can occur, but less than 0 fours cannot.


The confidence interval is plotted on the relative frequency figure with dashed black lines.  Hit "Roll" several times to draw new samples and see how often the estimate of the probability of rolling a 4 (or the probability of rolling any value) exceeds the interval.

Repeat this experiment for varying sample sizes of N = 100, 1,000, and 10,000 rolls. Inspect and consider how the width of the 90% confidence intervals changes for each sample size.

Question 3: Is the Normal approximation of the confidence interval adequate for the larger sample sizes?

N*p > 5 for all larger sample sizes, meaning the Normal approximate is an adequate approximation.

Step 3: Multiple Experiments

Navigate to the Multiple tab.

This section allows us to repeat the experiments an n number of times. A single experiment consists of rolling the dice an N number of times.

20 experiments. Enter 12 as the number of rolls and 20 as the number of experiments and click "Roll". This will perform 20 experiments of a dice rolled 12 times. Examine the resulting plot in the top-right panel (Probability of Rolling a 4 Across Experiments). Look at the width of the confidence bounds. Now, change the number of rolls to 100.

Question 4: How many experiments have a result outside the 90% confidence bounds? How many would you expect?

With 20 experiments, it would be expected that approximately 2 experiments (10% of the results) would lie outside the 90% confidence bounds.

Repeat the experiments by increasing the number of rolls to 1,000 and 10,000 and review the resulting confidence bounds in the plot. Feel free to try different combinations of sample size and number of experiments.

Step 4: Multiple Experiment Comparison

Navigate to the Multiple Comparison tab.

20 experiments. This section performs four scenarios, each with the same number n of experiments, but with different number N of rolls. By default, the number of rolls will be set to 12, 100, 1,000, and 10,000 and the number of experiments is 20. Click "Roll" to run the simulation. Examine the various results tables and plots. The top-right plot shows the average probability for each of the four samples (blue) and the standard error (red). The dashed black line is the expected (true) probability of 1/6. Hit "Roll" a few times to draw new random samples each time.

Question 5: How does the estimated probability and standard error differ across samples of different sizes?

The larger the sample size, the closer the estimated probability is to the true value (1/6). The standard error also decreases with increasing sample size.

1,000 experiments. Change the number of experiments to 1,000. View the bottom-right plot ("Plot"), which shows a histogram of estimated probabilities for each of the four scenarios.

Question 6: What is special about the estimates for sample size N=12?  What causes this result?

Estimated values of p are widely spaced (coarse).  This is because there are a limited number of estimates possible, computed by 1/12, 2/12, 3/12, …, 12/12.  Larger sample sizes have less coarse estimates.

View the bottom-right table, which tabulates the estimated probability of rolling a 4 for each scenario and for each experiment. The very bottom of the table, under all the experiment results, calculates the mean, standard deviation, and skew from the results of each scenario (each column of estimated probabilities). These summary statistics, which represent the sampling distribution for the estimated probabilities of rolling a 4, are also plotted in the "Statistics" tab. An example plot is shown below.

Question 7: What do the means say about relative frequency as an estimator of the probability of rolling a 4?

The means of the estimates are correct, even for N=12.  This means the estimator is UNBIASED.

Note that standard deviation decreases as sample size increases.

Question 8: What does the decreasing standard deviation with increasing sample size say about relative frequency as an estimator of the probability of rolling a 4? 

The fact that the standard deviations decrease with N means the estimator is CONSISTENT.