Posted by: W. E. Poplaski | October 6, 2008

Science: Data, Description and Experimentation

So, what do scientists do, anyway?

The easy answer is to say that scientists ‘do science’, which means it is the process of science that is held in common by all scientists.  This process is used to answer questions through the production and analysis of data.

What is the process scientists use to produce their data?  Generally, they collect their data using two powerful tools: carefully designed observational studies and experiments.

Observational Studies

The usual purpose of observational studies (also known as descriptive studies) is to describe phenomena—for example, its average value and the amount of its variability.  The study of obesity in adult males of a particular city would be an example of an observational study.  We would expect that study to report the mean amount of male obesity, the amount of variability in obesity, and how obesity varies (e.g., whether the distribution of obesity among adult males is normal, skewed, etc).

Descriptive studies can become quite complicated.  The obesity study might include separate results for young, middle-aged and elderly men; such a study would use stratified sampling.  Furthermore, several variables might be studied—e.g., blood pressure, blood cholesterol levels, amount of body fat, amount of daily exercise, daily caloric intake and amount of heart disease—and their averages and variability reported.

The study also might tell you how these variables are associated with each other.  That is, do men with greater than average amounts of body fat have higher blood pressure? Do these men tend to exercise more or less than men with less body fat?  Do they have more or less heart disease than those men with less body fat?  Scientists use correlation analysis to answer these questions.

Observational studies cannot tell you about the ‘cause and effect’ relationship between two variables.  For example, one cannot prove that smoking cigarettes causes cancer using an observational study.  Neither can one prove—using an observational study—that a high degree of obesity causes heart disease.  The best one can do with an observational study is to demonstrate the relationships among variables (e.g., high rates of smoking are associated with high rates of cancer). This is because an observational study cannot rule out the possibility that two variables (e.g., smoking and cancer) are linked together through an association with a third, as yet unknown, variable that causes the two original variables to vary together.

An example might shed some more light on this limitation.  An enterprising gentleman from the island of Nantucket collected this data on the local population over a six month period: weekly ice cream consumption and the weekly number of drownings.  He was surprised to notice that the correlation between ice cream consumption and drowning was strongly positive, meaning (he thought) that the more ice cream one consumed the greater one’s chance of drowning!  Was the consumption of ice cream causing drowning?  Alarmed, he took his evidence to the local public health officer.

The officer smiled and said, “Of course, it also might be that when the temperature gets hotter people tend to eat more ice cream and swim more in the ocean, than when the temperature is cooler.  More people swimming means more chances of drowning.”

Temperature, in the Case of the Deadly Ice Cream, was a lurking variable.  That is, it was an undetected variable whose relationship with the two variables of interest affected their relationship (i.e., their correlation with each other).  [When lurking variables are detected and identified, the partial correlation coefficient sometimes can be used to remove their effects.]

The problem with even well-designed observational studies is that there is no way to be certain they are free of lurking variables.  The descriptions these observational studies provide, nonetheless, are accurate and useful. However, determination of cause and effect is impossible. (Furthermore, even when lurking variables are absent, a correlation may be due to chance and not indicate a causal relationship.) Experiments must be used to determine causal relationships—e.g., between smoking cigarettes and cancer (or eating ice cream and drowning).


The key difference between an experiment and an observational study is that, in an experiment, treatments are randomly assigned to the experimental units or subjects.  This difference is what allows us to infer one variable has a causal relationship with another variable.

That means experiments always must have at least two variables: an independent variable and a dependent variable.  The independent variable is the treatment; the experimenter manipulates the level of this variable and observes the changes, if any, caused in the dependent variable.   The experimenter attempts to control or hold all other variables (except the dependent variable) constant so that they do not confound his results.

For example, our enterprising gentleman, skeptical of the Public Health Officer’s explanation, decided to do an experiment.  He planned to randomly select two groups of 50 Nantucketians.  Group A would be given an ice cream bar; Group B would be given a candy bar (the kind Nantucketians typically consume throughout the day).  One-hour after eating, the 100 Nantucketians (i.e., the experiment’s subjects) would swim fifty yards into the ocean and another fifty yards back to the beach—while rescuers on jet skis patrolled nearby (the jet-skiers do not know which group a subject belongs to.  That is to say, they are ‘blind’ to the subjects’ treatment assignment).  He reasoned that if ice cream were causing the drowning, then he should observe more Group ‘A’ than Group ‘B’ swimmers get into trouble.

Why are lurking variables not an issue, here?  At first glance it might seem that the answer is the ambient temperature does not vary because all 100 Nantucketians did the experiment at the same time.  Since temperature doesn’t vary, it cannot influence the outcome. True; however, even other undetected lurking variables are made unlikely as possible confounding influences.  This is because of the way the independent variable (the treatment) is manipulated by the experimenter—the experimenter randomly assigns the treatment levels (i.e., ice cream or candy) to the subjects.  This breaks any connection the treatment might otherwise have with another variable (e.g., swimming experience—if people who prefer ice cream are less experienced swimmers than those who abstain from the treat) and allows the experimenter to infer that variation in the independent variable is causing variation in the dependent variable.

Suppose our enterprising gentleman finds that 10 subjects from Group ‘A’ need to be rescued by the jet-skiers, but only 4 subjects from Group ‘B’ need rescue.  Would that be enough to demonstrate that ice cream causes drowning?

Not quite, because it is possible the experiment was not run properly or it may be that, by chance, treatment ‘A’ got more weak swimmers than treatment ‘B’.  He needs to replicate the experiment to conclusively demonstrate a difference between the two treatments.  That means, on another day, he should repeat the experiment with a different set of 100 subjects.  It is unlikely that by chance alone, again, those swimmers assigned treatment ‘A’ are weaker than than those assigned treatment ‘B’.  Depending on budgets and resources, many scientists prefer to perform their experiments at least three times (i.e., three replications) and average the results of the replicates.

Ultimately, the experiment needs to be replicated by a different experimenter for us to be completely confident that the results indicate causation.  Why? Perhaps our enterprising gentleman was sub-consciously introducing a bias into the experiment that favored one treatment over another.

So, the three great principles of experiments—random assignment of treatments to subjects, control of all other variables, and replication—are what enable experimenters to infer ‘cause and effect’.

Sometimes it can be difficult to tell whether an investigation is an observational study or an experiment.  When in doubt, ask yourself if the levels of at least one of the variables studied are randomly assigned to the subjects.  If so, then it likely is an experiment; if not, it is an observational study.


This website gives more details about experimental design:

Statistics for Experimenters by Box, Hunter & Hunter is an excellent book on the subject and not very technical—


If you are hungering for more information on causation, then view Prof Judea Pearl’s (UCLA) PowerPoint presentation on causation (it will take you from Adam and Eve to the twentieth century):



  1. […] also Science: Data, Description and Experimentation Possibly related posts: (automatically generated)aquarium […]

  2. […] variable is manipulated by the experimenter and its different levels are called treatments.  (See Data, Description and Experimentation for more about explanatory and Response […]

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: