Dr. Patrick McCarthy's
Notes on Research Methods
in I/O Psychology

These are some of my notes on research methods in I/O psychology. Actually, the methods and issues described below are relevant to all areas of psychology (not just I/O). You'll find that these notes can supplement your reading and, with any luck, help clarify some key points.

The notes below first outline key steps in the research process, then outline a few of the major types of research methods--and highlights advantages and disadvantages of each. Finally, we'll cover a few key issues in the measurement and statistical analysis of research data.

Steps in the Research Process

1. Determining What Topic to Study

• What is the general topic of interest to be studied?
• Question/curiousity you are trying to answer or problem you are trying to solve
• May come from own interests, or stimulated from other research

2. Development/Adoption of a Theory

• Theory expresses beliefs about the relevant behavior(s) of interest
• Proposes description & explanation of behavior(s), including key influencing factors
• Theory scientific only if it
• can be tested (*critically important)
• accounts for currently known facts
• enables predictions about future events
• Theory may develop inductively to summarize & explain data/facts already observed
• Theory may come from own ideas & logic (possibly including inductive inferences), then be used deductively to generate specific, testable hypotheses

3. Generation of Hypotheses

• A hypothesis is a statement about the expected relationships between variables
• Variables are the things/events that the research investigation measures
• Operational definitions of variables are critical
• Each hypothesis will be tested from analyses of the research data collected
• (note: plural of "hypothesis" is "hypotheses": i.e., one hypothesis, multiple hypotheses)

4. Selection of an Experimental Design

• Research setting & degree of control over relevant variables guides the methods chosen to conduct the research
• Major types of research methods, including advantages/disadvantages of each, are outlined later in this document

5. Collection of Data

• Particulars of data collection depend on experimental design selected
• Sampling is critical issue for data collection in all instances
• How representative is sample of population want to generalize to?
• Results only generalizable if have representative sample
• Random sampling usually first choice
• All from population have equal chance of being chosen to be research participant
• Stratified sampling sometimes used
• Selection of research participants based on categories that represent important distinguishing characteristics of a population
• e.g., decide that since 40% of workers female, want 40% of participants to be female

6. Statistical Analysis of Data

• Statistics are tools used by the researcher to help make sense of the data collected

7. Interpretation of Results & Drawing Conclusions

• Researcher identifies and describes the key findings
• Some important considerations:
• What were the most important findings? And why are they so important?
• To whom are the findings most relevant?
• What are the limitations of the research?
• What additional (follow-up) research is suggested by the findings?

8. Further Evaluation & Replication of Findings

• One person's findings in one study is not enough to conclusively establish findings
• Additional related research, including via alternative methods, is needed
• Consistency of pattern across studies is crucial component for findings/conclusions to be confidently accepted by scientific community

 

Research Methods

The major types of research methods vary in terms of the experimenter's control over relevant variables, and the naturalness of the setting. These differences have a direct bearing on critical issues, such what sorts of conclusions we can make from the findings, and how confident we are of the validity of those conclusions.

You will find the word "experiment" is reserved for specific circumstances. Experimental methods are those in which the experimenter has control over: a) the assignment of subjects to conditions (usually through use of random assignment), and b) the manipulation of the independent variable. If either of these two are not present, then we call it a research study.

Laboratory Experiment

• Advantages
• experimenter has greatest degree of control over variables of influence, thus
• is best method for inferring causality
• measurement of behavior most precise
• can most easily replicate since most easily account for experimental conditions

 
• Disadvantages
• laboratory may lack realism
• some phenomena can't be studied in laboratory
• e.g., how riots affect social attitudes
• limited generality of findings to the "real world"

Field Experiment

• Advantages
• if proper experimental design, can suggest causal inferences
• realism good because in natural setting
• thus generalization of results good
• can address impact of complex behaviors in real-life contexts

• Disadvantages
• less control
• sometimes people refuse to participate
• difficulty getting access to a business or industrial setting for the experiment

Field Study

• Advantages
• high degree of realism because are in natural environments
• data on large number of variables can be collected at the same time
• researcher doesn't have as great an impact on the study as he/she may in other strategies

• Disadvantages
• variables not manipulated by the researcher
• thus, typically unable to infer causality
• companies may not give permission to conduct field study
• measurement of variables less precise than in laboratory

Survey Study

• Advantages
• data collected in naturally occurring environment
• results of surveys often yield new hypotheses to test by another method
• several survey techniques available
• e.g., questionnaire, interviews, observation

• Disadvantages
• little or no control over variables in the study
• inferences made only through the use of complex statistical analyses
• low return rates threaten representativeness of responses, thus biasing the results
• what people say doesn't always agree well with what they do

Case Study

• Advantages
• provides detailed description of specific example
• e.g., business practices at a particularly successful company
• e.g., leadership style of an especially respected leader
• may spark development of hypotheses for later testing by other methods (in experiments or studies)

• Disadvantages
• cannot draw firm conclusions about generalizing findings beyond the specific example
• what worked in the one example studied may not work elsewhere
• cannot test hypotheses
• usually cannot establish cause-effect relationships

 

Measurement & Statistical Analyses of Data

Key Issues in Measurement

• e.g., if I take an IQ test more than once, will my scores be consistent or vary wildly?
• (there are various specific types of reliability)

• Validity can be divided into two main subtypes:
• internal validity is whether the test really measures what it claims to (or whether something else--like other abilities or the situation--influence a person's score)
• external validity is: to what degree can the finding of this specific study be generalized to other people and places?

• Descriptive statistics summarize data, and include: frequency distribution, mean, median, and standard deviation (B366 students: know each of these from your reading)

Inferential Statistical Analyses of Experimental Data

• Inferential statistics are analyses of data that directly test hypotheses
• Statistical significance means that the results likely did not occur simply by chance
• t-test is a common inferential stat for when comparing two groups' scores on a dependent variable
• Analysis of variance (ANOVA) is sort of like a t-test, but is used when comparing three or more groups on a single dependent variable
• Multivariate analysis of variance (MANOVA) is used when comparing multiple groups on multiple dependent variables

Correlational Method

• typically no control over variables
• field studies and survey studies are common specific types of the correlational method
• can reveal degree to which two or more variables are related, but does not establish the cause(s) for such relations
• correlational statistics used to describe strength of relations between variables:
• Pearson Product Moment (r) -- for correlation between two variables, one independent variable & one dependent variable
• also known as a "bivariate relation"
• Multiple regresssion (R) -- for correlation between two or more independent variables, and one dependent variable
• known as a "multivariate relation"
• Structural equation modeling -- describes how well a predicted pattern of relations among a set of variables is described by actual data/measures of those variables
• Factor analysis -- reveals consistencies in data from several variables, enabling organization of many variables into a few summary "factors"
• Meta-analysis -- is a technique for getting a single summary correlation statistic of the results from several different previous studies
• is relatively new (emerged in the 1980's) and somewhat controversial

• Here is where you can return to Dr. McCarthy's Course Supplements webpage
• Or you can go to his Homepage, if you like