Posted by: W. E. Poplaski | March 2, 2009

THE COMPLEAT EXPERIMENTER: 2. Types of Mistakes

THE COMPLEAT EXPERIMENTER

~ ♦ ~ ~ ♦ ~ ~ ♦ ~

O, sir, doubt not that experimenting is an art; is it not an art to tease out the native hue of resolution?

~ ♦ ~ ~ ♦ ~ ~ ♦ ~

2. Types of Mistakes

The experimenter chooses her explanatory and response variables, does her experiment (controlled, randomized and replicated), but when she analyzes the data, she sees no difference between the treatment levels of the experimental units. Was the experiment necessarily a failure?

No! It may be that the experiment accurately describes the situation and the putative “explanatory” variable has no effect on the response variable. She has gained new knowledge and now knows to look elsewhere for causes of change in the response variable.

So, can experiments go wrong even when they are properly controlled, randomized and replicated?

Yes. But first, let’s look more closely at the workings of a simple experiment. This experiment has the explanatory variable at two levels or settings: control and experimental. The experimenter measures the response variable on the experiment’s units and compares the results for the two treatment levels. She does this by defining a null hypothesis and an alternative hypothesis.

The null hypothesis—or hypothesis of ‘no difference’—is that the measurements of the response variable are the same for both the experiment’s units receiving the control and those receiving the experimental treatments. The alternative hypothesis is that there is a difference in the response variable between the experiment’s units of the two treatment levels. She may make either one of two types of mistakes regarding these hypotheses.

When there is no difference between the two treatment levels, the experiment can either correctly not detect a difference or incorrectly detect a difference. In the later case, the experimenter incorrectly rejects the null hypothesis mistakenly affirming that the explanatory variable did cause a change in the response variable (when it did not). We describe that mistake as a ‘Type I Error’. It also is known as an error of ‘false hope’ because we end up thinking that our experimental treatment is better (or different) than the control, when it is not. It is an error because we failed to choose the null hypothesis (when the null hypothesis is, in fact, the correct choice).

When there is a difference between the two treatment levels, the experiment can either correctly detect that difference or not. If not, then the experimenter incorrectly accepts the null hypothesis mistakenly affirming that the explanatory variable did not cause a change in the response variable. We describe that mistake as a ‘Type II Error’.  It is also known as an error of ‘missed opportunity’.

(See https://wepoplaski.wordpress.com/2008/07/09/distributions-hypotheses-and-guppies/ for more information about Type I and Type II Errors.)

So, a Type I error occurs when the experimenter sees a difference that is not there (false hope), and a Type II error occurs when the experimenter fails to see a true difference (missed opportunity). Those are the only mistakes an experimenter can make in drawing a conclusion from his or her experimental results!

See The Compleat Experimenter: 3. Experimentation and Power.

The Compleat Experimenter: 1. What is an Experiment?

Advertisements

Responses

  1. […] Next, The Compleat Experimenter: 2. Types of Mistakes. […]


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: