SURVEYS
I. Introduction to survey research.
A. Survey research methods collect self-reported data from individuals and, occasionally, small groups. They are most often used to measure the characteristics, attitudes, and behaviors of individuals, but may also be used to collect data about small groups or larger organizations. By collecting data from large, representative samples of individuals, surveys can even be used to draw conclusions about institutions, cities, communities, or entire societies.
B. The popularity of surveys.
C. Types of surveys.
1. Self-administering questionnaires.
a. Designed so that respondents can read and complete it on their own with little or no involvement from the researcher.
b. Typically they are mailed to respondents, delivered by hand to individuals or groups, or made available over the internet.
2. Interviews.
a. Requires personal contact between researcher and respondent.
b. Interviews can be conducted by going to people's homes or places of business, by telephone, or even online.
II. General guidelines for constructing questions in survey research.
A. Keep questions simple and clear.
1. Avoid long and wordy questions.
2. Avoid jargon and slang.
3. Avoid confusing and vague words.
4. Avoid negative items.
5. Avoid double-barreled questions.
B. Make sure you ask questions your subjects can answer.
1. Avoid using words your subjects won't understand.
2. Avoid asking questions your subjects will not be able to answer or that don't apply to your subjects.
3. Avoid asking questions your subjects have no interest in.
4. Avoid false premises, hypothetical situations, and speculative questions.
C. Avoid biased or leading items.
1. Leading questions.
2. Biased wording.
3. Social desirability.
III. Types of questions.
A. Closed-ended/fixed choice/forced choice questions.
1. Strengths.
a. Predictable and reliable.
b. Easily coded and quantified.
c. Useful in composite measures.
d. Easy to answer.
2. Weaknesses.
a. May lack validity.
b. Limits the kind of information you get.
c. Restricts respondents' answers.
3. Answer categories must be exhaustive and mutually exclusive.
4. Types of closed ended questions.
a. Yes/no questions.
b. Likert-type items.
c. Semantic differentials.
d. Multiple-choice questions.
e. Check lists.
f. Rank order questions.
B. Open-ended questions
1. Strengths.
a. Increases variability.
b. Increases validity.
2. Weaknesses.
a. Less reliable.
b. Difficult to code and quantify.
c. Often left unanswered.
3. Following closed-ended questions with open-ended questions.
4. Coding open-ended questions.
5. As a general rule, limit open-ended questions.
C. Contingency questions.
1. Simple contingency questions.
2. Complex contingency questions.
3. Problems with contingency questions.
D. Matrix questions.
1. Types of matrix questions.
2. Matrix questions and composite measures.
3. Response set.
4. The benefits of matrix questions.
IV. Developing a self-administering questionnaire.
A. Questionnaires should look attractive, professional, and interesting.
1. You want respondents to be interested enough to complete the questionnaire.
2. You want them to take it seriously.
B. Make sure it is clear and legible.
1. Make sure everything is spelled right and your grammar is correct.
2. Don't crowd things together. Make it easy to read, but don't waste space. You don't want it to appear to long.
3. It should be typed on a quality typewriter or printed on a quality printer.
4. All copies should be good quality.
C. Question order is important.
1. You want your questionnaire to be logical, connected, and consistent. It should "make sense" to those completing it.
a. Put related questions together.
b. Use logical groupings with headings.
2. Try to put interesting questions first, but if some questions are particularly troubling or threatening, you should put them in the middle.
3. Generally put demographic questions near the end.
4. Question order, interpretive context, and validity.
D. Give clear instructions.
1. General and specific instructions.
a. The cover letter/instruction page.
b. Instructions in the body of the questionnaire.
2. Make sure you tell respondents how and where to answer questions.
a. Make it as easy as possible.
b. Be as consistent as possible.
c. Using a separate answer sheet.
3. Pay careful attention to changes in the type of questions or the way answers should be marked.
4. Pay careful attention to contingency and matrix questions.
5. Put instructions for returning the questionnaire on the instruction page and at the end of the questionnaire.
V. Distributing self-administering questionnaires via the mail.
A. The cover letter.
1. State the nature and purpose of study clearly and simply.
2. Tell respondents exactly what is expected of them and what they can expect in the questionnaire.
3. Request participation, making sure respondents are aware that their participation is voluntary.
4. Tell respondents if the questionnaire contains questions that might be troubling or sensitive, and tell them if they find any questions troubling they may skip them.
5. If you are providing a referral, put that information in your cover letter.
6. If the study is anonymous tell them so and what is being done to maintain anonymity. If it is not anonymous, tell them why it is not and what is being done to protect their confidentiality.
7. Provide an accurate time estimate.
8. If you are offering any kind of incentive for returning the questionnaire, discuss it in the cover letter.
9. Express your appreciation in advance.
10. Remember, always be polite and courteous, but to the point.
B. Instruction page.
1. Provide a general discussion of how answers are to be marked.
2. Discuss each type of question and how it is to be answered.
3. If there are large blocks of questions that only apply to certain groups, that should be pointed out. Give instructions on how to follow contingency questions.
4. If a separate answer sheet is used, discuss how it is to be marked.
5. Tell them there also will be instructions in the body of the questionnaire.
6. Include instructions on how to return the completed questionnaire.
7. Incorporating instructions into the cover letter.
C. Return envelope and postage.
1. Make it easy for them to return the questionnaire.
2. Include a self-addressed stamped envelope.
3. Alternatives to the self-addressed envelope.
D. Practical matters.
1. Mailing options.
a. Special delivery.
b. First class.
c. Bulk mail.
d. Business return.
2. Postage regulations.
a. Size and weight
b. Packaging.
3. Timing.
a. Avoid mailing surveys so that they arrive around holidays or vacation periods.
b. Shortly after the first of the year is a good time.
E. Response rates.
1. Responses rates have to do with the proportion of surveys that get returned.
a. Low response rates threaten the representativeness of samples.
b. Generally, a 50% response rate is generally considered adequate if sample size was large enough.
c. A 60% response rate is considered good.
d. A response rate of 70% or more is very good.
2. Improving response rates.
a. The importance of appearance.
b. Appeals and inducements.
c. Keep things simple.
d. Keep instructions clear.
F. Receiving questionnaires.
1. Previewing, numbering, and recording.
a. Response graphs.
b. Identifying non-responders.
2. Follow up.
a. Letter only follow-ups.
b. Letter and a new survey.
c. Follow up phone calls.
d. No more than 2 follow up mailings two or three weeks apart.
VI. Distributing self-administering questionnaires by hand.
A. Door to door delivery.
B. Distribution to groups.
C. Having others assist in the distribution.
D. General guidelines for hand delivery.
1. Always appear serious and professional with respondents.
2. Make sure oral explanations and instructions are clear.
3. Have a clear strategy for collecting surveys.
E. While hand delivery can increase response rates, it can be more costly and more time consuming, especially if large representative samples are used.
F. Anonymity, confidentiality, and informed consent.
VII. Developing and administering electronic/online questionnaires.
A. Most everything above about developing self-administering questionnaires applies to electronic/online questionnaires as well, but there are some unique
issues.
1. Questionnaire construction.
a. Software.
i. Word processing and PDF software.
ii. Web authoring software.
iii. Specialized online survey software.
b. General structure.
i. Single page document respondents will scroll through.
ii. Multi-page document with each section on a separate page.
iii. One question at a time.
c. Answer types.
i. Radio button (select only one).
ii. Check boxes (select more than one).
iii. Drop down menu (for long lists of answer choices).
iv. Single and multi-line text boxes.
v. Numerical data boxes.
c. Dealing with varying platforms and web browsers.
d. Use of color, graphics, sound, and video.
e. Cover page/introduction.
f. Instructions and the importance of consistency.
2. Administration/distribution.
a. Emailing the questionnaire.
b. Posting the questionnaire on the web.
B. Advantages of electronic/online questionnaires.
1. Collect large amounts of data quickly and inexpensively.
2. Automation.
a. Automating contingency questions and controlling navigation through the questionnaire.
b. Checking and verifying responses.
c. Automatic data entry.
C. Disadvantages of electronic/online questionnaires.
1. Requires subjects have access to and knowledge of computers and the internet. Can bias the sample toward younger, more educated
respondents.
2. Technical problems and glitches.
3. Limited control over who has access to and completes questionnaire.
VIII.
Conducting face-to-face interviews.
A. Designing the interview schedule.
1. The interview schedule is developed before interviews are actually conducted. It is similar to the self- administering questionnaire except it is addressed to the interviewer.
2. The interview schedule should contain:
a. Instructions for the interviewer as to how to conduct the interview.
b. A list of all the questions to be asked, instructions for asking them, and a list of the available answer categories.
3. It might also contain:
a. Information about permissible specifications, clarifications, and probes.
b. Guidelines for dealing with problems that might come up during the interview.
4. Questions in the interview schedule.
a. The same types of questions used in a self-administering questionnaire can be used in an interview, but some may require slight modifications.
i. Closed-ended questions.
ii. Open-ended questions.
iii. Contingency questions.
iv. Matrix questions.
b. As in the self-administering questionnaire, keep questions and answers simple and clear.
5. Questions order.
a. Putting demographic and "get-to-know-you" questions first.
b. Put threatening or challenging questions near the end.
c. Use logical groupings.
d. Make the interview schedule logical, connected, and consistent.
6. Marking and recording respondents answers on the interview schedule.
7. Make sure the interview schedule is neat and easy to follow.
B. General guidelines for conducting interviews.
1. Familiarize interviewers with the interview schedule.
2. Conduct practice interviews.
3. Maintain a pleasant appearance and demeanor.
a. Dress professionally, but similar to interviewees.
b. Be neat and clean.
c. Be pleasant and friendly.
d. Minimize personal qualities to reduce interviewer effect.
4. Be an interested, yet neutral listener.
C. Specific guidelines.
1. Convincing the people to be interviewed.
a. Others they know have participated.
b. Their opinion is highly valued.
c. The project is important.
d. Personal appeals.
2. Establishing rapport.
a. Being open and friendly.
b. Establishing trust.
c. Casual conversation.
3. Asking Questions.
a. Follow the interview schedule exactly.
b. Try to make the interview seem as much like natural conversation as possible. Make the transition and flow from one question to another smooth and logical.
4. Recording Responses.
a. Allowing them to come up with their own answer then checking the most appropriate answer on the interview schedule.
b. Reading answer choices.
i. Reading a list of possible answers and having respondents stop the interviewer when correct answer is read.
ii. Reading all available choices then having respondent select the best one.
iii. Using reminder cards.
c. Marking fixed choice answers.
d. Recording open-ended responses.
5. Expressing thanks and appreciation.
D. Ethical issues.
1. Informed consent and voluntary participation.
2. Sensitive questions.
3. Anonymity and confidentiality.
E. Coordination and control of multiple interviewers.
1. Training.
a. Explain the study.
b. Offer general guidelines.
c. Go through the interview schedule.
i. Ask for questions or problems.
ii. Provide probes and specifications.
d. Demonstrate and practice interviewing.
2. Supervision and evaluation.
a. Following up and checking.
b. Contacting interviewees.
F. Advantages of face-to-face interviews over questionnaires.
1. More personal and perhaps more valid.
2. Better response rates.
3. Decreases the number of "don't know" answers.
4. May allow extra explanation and clarification.
5. Observations can be made.
G. Disadvantages of face-to-face interviews.
1. Lack of anonymity.
2. Interviewer effect and social desirability.
3. Coordination and control of multiple interviewers.
4. Reliability of multiple interviewers.
5. More time consuming and costly.
6. More risky.
IX. Telephone interviewing.
A. The increasing popularity of telephone interviewing.
B. Computer assisted telephone interviewing.
1. Random digit dialing.
2. Teleprompting.
3. Automatic data entry.
C. Advantages of telephone interviews.
1. Sampling through random digit dialing.
2. Easy, quick, safe, and inexpensive.
3. Can be anonymous.
4. Reduces interviewer effect.
5. Easier to supervise.
D. Disadvantages of telephone interviewing.
1. Only people with phones who happen to be at home when you call can be in study.
2. Unlisted numbers and invasion of privacy.
3. Non-household numbers.
4. Sampling within households.
5. Bogus interviews and telephone solicitations.
6. Ease of termination.
X. The advantages and disadvantages of survey research over other methods.
A. Advantages.
1. They allow the collection of large amounts of data from large numbers of individuals in a relatively short period of time.
2. Existing measurement instruments can be incorporated easily saving time and increasing reliability.
3. Multiple and composite measurements are easy to incorporate increasing validity and allowing the measurement of complex and multidimensional concepts.
4. Many variables can be measured simultaneously, allowing complex causal models to be investigated and facilitating statistical control.
5. Data are typically easy to code and quantify for statistical analysis.
6. They can easily be administered to representative samples, allowing statistical inference to larger populations.
7. Self-administering questionnaire and telephone surveys are relatively safe, fairly easy to carry out, and time efficient. (Face-to-face interviews can be more risky and time consuming.)
8. Self-administering questionnaire and telephone surveys can be conducted anonymously and pose few ethical risks. (Face-to-face interviews obviously cannot be anonymous.)
B. Disadvantages.
1. They are limited to self-reported data. (Limited direct observations sometimes can be made in face-to-face interviews.)
2. They generally are limited to individuals as the unit of observation.
3. They do not allow the careful controls that experiments do, making it more difficult to demonstrate time order and to evaluate causal relationships.
4. They have some difficulty addressing change over time.
5. They can be expensive if done on a large scale.
6. They are dependent on representative samples, problems arise if sampling is done incorrectly or response rates are low. (Face-to-face interviews can significantly increase response rates.)
XI. Steps in designing a survey research project.
A. Choose a topic, review the literature, and develop a research question or questions.
B. Clearly state your hypothesis or hypotheses and identify the key variables.
C. Operationalize the dependent variable, the independent variables, and any intervening variables.
D. Develop a questionnaire or interview guide.
E. Draw a random sample.
F. Distribute questionnaires or conduct interviews, insuring that human subjects are adequately protected.
G. Analyze results and draw conclusions about hypotheses.
H. Publish findings and conclusions.
XII. Using surveys in applied research.
A.
Needs assessment.
B. Program evaluation and outcome assessment.
C. Client, employee, and product satisfaction research.
D. Cost-benefit analysis.
E. Social impact assessment.
F. Action research.
G. Operations research and organizational analysis.
H. Market research and utilization studies.
I. Public opinion and political polling.
J. Quality assurance research.