Experiments provide a way to test how a variable affects a situation. It is important to design the experiment carefully so that other variables that might have an effect on the response are accounted for.

One important aspect in experimental design is the use of randomness to assign groups for the experiment. Random assignments are used so that there is less of a chance that other variables will affect the data.

For example, a researcher wants to study the question, “Does the size of the aquarium in which frogs are kept affect the sizes of the frogs?”

The researcher selects 20 young frogs to be used in the experiment. The frogs are then numbered, and a random number generator is used to separate the frogs into two groups of 10. One group will be put in a 10-gallon aquarium to grow, while the other group is placed in a 100-gallon aquarium to grow. The researcher will measure the sizes of the frogs, using their weight, at the end of a year.

The size of the aquarium is one of the variables, and a treatment is the value of the variable that is changed between the two groups in an experiment. In this experiment, there are two treatments—a smaller aquarium and a larger aquarium.

The researcher finds that the frogs in the smaller aquarium have a mean weight of 111.2 grams and the frogs in the larger aquarium have a mean weight of 169.3 grams. The difference of 58.1 grams seems large, but is it enough to say the aquarium size is the cause of the difference? Even if all the frogs were in the same aquarium, we expect there to be some variability in the weight of the frogs. If some of the frogs were switched at the beginning of the experiment, would there still be a large difference? A simulation can be used to see if differences we see in our groupings are also consistent with groupings that did not depend on our treatment.

In this kind of simulation, all the data is grouped together, then data is randomly redistributed among two groups, and the difference between the means of these new groups is computed. This process is repeated several times to create a randomization distribution: a distribution of the differences between the means for the treatment groups containing randomly redistributed data. The mean difference from the experiment is then compared to this distribution to determine whether such differences are also consistent with groupings that did not depend on the treatment. To investigate whether the aquarium size is the important factor in the difference or whether the difference could be due to the chance ways the frogs were separated into treatment groups, we can use a simulation to examine what results we might expect from chance. For the moment, we assume the aquarium does not play a part in the size of the frogs, and we put all 20 frog weights into one group. We can separate the weights into two groups in a random way and determine the difference in mean weights between the groups.

Doing this many times will produce a distribution of weight differences that could be the result of how the frogs happened to be divided into groups. We can then compare the actual difference in mean weights from the treatment groups to the randomization distribution to determine if the 58.1-gram difference is unusual (which would suggest the aquarium size played an important role) or whether this difference is also typical of what we might see from random assignment to groups.

Randomly breaking the data into two groups 30 times produces the histogram here.

Notice that, based on this distribution, a difference of either 58.1 grams or greater or -58.1 or less is unusual. This suggests that the original difference of 58.1 grams is highly unlikely to occur if the aquarium size played no role. The researcher has evidence to support the hypothesis that aquarium size has an impact on the weight of frogs growing in it.

If the original difference had been about 15 grams, there would have been reason to believe that the weight difference observed might be due to random chance because 11 out of the 30 differences in the randomization distribution are at least 15 grams away from 0.