Langston, Research
Methods, Notes 14 -- Non-experimental designs
I. Goals.
A. Kinds of single-N (small-N) designs.
B. Methods.
C. Quasi experiments.
D. Pros and cons.
II. Kinds of single-N (small-N) designs. These are
technically experiments with only one participant, but we'll be a little
more general and include experiments where you have so few participants
that you only have one group (no control group).
A. Case studies: A source of rich information about a single
individual. Find out everything you can about one case, try to find
some general principles that could be applied to other people with similar
problems (Freud, Skinner).
B. Single-N experiment: Do an experiment with just one participant.
C. Small group: An experiment like B, but with a group of
participants, no control group.
Top
III. Methods.
A. Baseline designs:
1. Simple: Measure, treat, measure (AB design). Schematically:
First, you take a baseline measure. Then you treat. Then you
look for changes from baseline. If performance goes up (or down) after
treatment, you want to conclude that the change was caused by the treatment.
You can see the inherent weakness here pretty quickly. The difference
could be due to chance variation (performance might make lots of big changes
from time to time, and you just picked up one of these regular fluctuations),
or some other event (a confound) might have occurred that really made the
change. This is bad.
2. Complex: Measure, treat, measure, withdraw treatment, measure...
(a.k.a. ABAB or reversal design). Schematically:
First you get a baseline. Then, you treat and look for a change.
Then you remove treatment and look for a return to baseline. Then you
treat again. The idea is that if performance goes up when you add treatment
and down when you remove it, the treatment is causing those changes.
Problems:
a. Ethics: Can you remove a treatment that works from a patient
who needs it?
b. You can only do this with something that will return to baseline
when you remove treatment. So, if the treatment has a lasting effect,
this won't work.
B. Multiple baseline: Introduce treatment at different times
for different aspects of a situation and look for change only after treatment,
regardless of when it's introduced. Schematically:
1. Individuals: Each graph (A, B, C) represents a different
individual (who are ideally all in the same situation, like the same mental
institution). You introduce treatment for A earlier than B, and B
earlier than C. If each person only improves after treatment, it's
probably not a confound causing the effect (if it was a confound, then B
and C would have improved at the same time A improved).
2. Behaviors: A, B, and C could also be three behaviors in
one person. For example, a person might have obsessive thoughts about
the stove being on, the baby falling in the toilet, and a counting ritual.
You could introduce therapy for each obsession at a different time to demonstrate
that the therapy is effective for solving the person's problem.
3. Situations: If a person is overly aggressive with family,
friends, and coworkers, then A, B, and C are three situations. You
could introduce therapy for aggression in each situation, and show that it's
the therapy that improves relations.
Top
IV. Quasi experiments. Something about the design can
compromise internal validity and makes causal conclusions difficult.
What is that something? No random assignment.
A. Ex post facto (after the fact). You look at the effect
of some intervention or event on people. For example, you could look
at the effect of a hurricane on people's mood. The design can look
a lot like an experiment: Take people who were in the hurricane and
some who weren't, compare mood. You can even match these people on some
variables to make sure they're similar. But, since you didn't randomly
assign and manipulate the presence of the hurricane, you can't tell for sure
that differences were caused by the “IV.”
B. One-shot case study: You get the case after the “treatment”
has occurred. So, you can't get a baseline to compare to. For
example, you get an alcoholic with memory problems. You can select
a comparison person (same age, profession, social class, etc.) and compare
the two, but without random assignment, it's all suspect.
C. Interrupted-time-series. An extended AB design where you
didn't introduce the treatment. For example, I might hypothesize that
heat increases violence. I could measure violence in the winter, and
again in the summer. If I go for several years, it could be an ABAB
design.
D. Participant variables. You want to “manipulate” something
like sex of the participant. All of the correlation confounds apply.
You can try to match, but it's hard, and probably not entirely possible.
Top
V. Pros and cons.
A. Pros:
1. Source of ideas: Use these to suggest experiments to perform.
2. Therapeutic innovation: If you get an interesting case,
study it carefully, try various therapies, see if you can come up with some
general principles to apply to other individuals.
3. Study rare phenomena: If you only get one neuropsychological
case of a patient who loses the ability to name fruits after a head injury,
then that's all you have to study.
4. Challenge assumptions: If a theory says “If p then q” (like
“If you have bulimia, then you've repressed memories of childhood sexual
abuse”), all you need is one case of a person with bulimia who isn't repressing
memories of sexual abuse to prove the hypothesis wrong.
B. Cons:
1. Cause-effect relationships are hard to determine. With
no random assignment and no control group, you're never really sure if changes
are due to treatment or a confound.
2. Biases: Confirmation biases can influence data collection.
Plus, the experimenter is usually a participant in the research situation
(like the therapist), which introduces numerous other sources of bias.
3. Generalizability: Can you really make conclusions about
a population on the basis of a single case?
Conclusion: There are problems with these designs, but in situations
where it's your only option, they are better than nothing.
Top
Research Methods Notes 14
Will Langston
Back to Langston's Research Methods Page