Activity #7

In the following, Jim Friedrich describes a couple activities that help students to learn about proper experimental design and hypothesis testing. The first activity, in particular, makes students aware of the need to control for expectancy effects on the part of both experimenters and participants.

Fri, 18 Aug 2000
Jim Friedrich
Willamette University
jfriedri@willamette.edu

I tell students about a friend's belief that our bodies have "auras" that are disrupted when they come in contact with harmful organic substances. One prediction is that even holding a cigarette will weaken the body. I ask for about 10 volunteers (usually just calling the people in the first few rows). I tell them we will follow a very standardized procedure and to hold all comments/critiques until we are done.

A student participant is asked to stand next to a table in the front of the room. S/he is asked to face the class, holding nothing, and to extend the left arm parallel to the floor and out to the side. I stand behind them and tell them I will be trying to push their arm down and that they should resist any movement downward as hard/long as they can. I place my two index fingers together side by side about where a wristwatch would be on the person's arm and push down for about two seconds == hard but not so hard that their arm actually collapses (*it usually just wobbles and drops an inch or so). Then I give them a cigarette to hold in their free right hand and repeat the procedure

Participating students are asked to mentally note which trial made them the weakest. Students in the class serve as observers and record in which case they felt the arm appeared to go down more easily. The results tend to be rather dramatic; the arm almost invariably collapses in the second trial, and participants don't report a sense of greater pressure (nor do i apply any). Initially skeptical participants often have stunned looks on their faces, and the class is pretty focused (and unanimous in their judgments!).

After this initial amazement, we settle down to examining design problems, e.g.
counterbalance for order (possible fatigue effect?)
subject expectancy (participants know the hypothesis?)
experimenter bias (do i really apply consistent pressure across trials?)
proper control condition (is it simply a matter of holding anything at all?)

We discuss what could be done to remedy each of these problems and improve the design. We then talk about what could be concluded if these changes were made and the result persisted. (Have we examined mediating variables specified by the theory of auras? Can one ever prove a theory true?)

As it turns out, this demonstration also lends itself well to talking about statistical significance testing. When I use it in statistics, I follow the demonstration with a discussion of whether the results (e.g. the cigarette arm going down easier say 90% of the time) could easily occur by chance. We model this with a coin tossing procedure. I supply students with cups holding ten pennies each. We take about 100 samples of N=10 (e.g. in a class of 50, each student would pour out the cup twice) and record the number of heads on the board in a histogram (an empirical sampling distribution). It is quickly apparent that if arms are equally likely to be weaker with or without the cigarette, having 9 of 10 trials "favor" the cigarette arm is about as unlikely as getting 9 heads in a sample of 10 (rare in the sampling distribution!) We use this to talk about what "p<.05 " means. More importantly, we link this to the design problems to point out that the statistical test only documents a reliable association. The results are indeed statistically significant -- there appear to be non-chance differences in performance between the two conditions. But what accounts for that difference (aura effects, fatigue, subject or experimenter expectancy, etc) cannot be determined by significance testing. It instead is a function of the proper design of a study.