Tom’s Ten Data Tips – January 2010
Survey design
Survey data are a useful complement to electronic registrations of behavior. They allow you to relate opinion to actions. Survey design choices impact your results, and these measurement effects ought to be mitigated. You can never take survey data at “face value” without an understanding of how the sample was selected and how subjects were interviewed.
1. Begin With Research Objectives
Survey design should always begin with, or rather be preceded by unambiguous definition of research objectives. After you have formulated exactly what questions your study should answer, you define a sample (see also a previous newsletter on Survey Sampling), data collection method(s), and analysis techniques.
Objectives should drive the research method, and not the other way around. Focus on the business decisions you are trying to support, and determine the minimally sufficient inputs. Avoid gathering peripheral information just because you are surveying anyway. Every single question needs to relate to your objectives, or else it should be dropped.
2. Translate Research Objectives Into Information Requirements
Decisions you are trying to support have a direct bearing on the research objectives. And from the research objectives follow your information requirements. During briefing these should be surfaced, along with (hidden) assumptions being made. The debriefing (in writing!) should contain both these translations, along with a high-level (cost) justification for the implementation choices being made. This needn’t be a detailed business case, merely a demonstration that your research design is feasible, along with a summary of most important assumptions.
Incidentally, it is exactly this “translation stage” which requires an open mind, that might trigger alternative methods as an adjunct or substitute altogether (see also tip# 10).
3. “Lead” Respondents Into Your Survey With “Easy” Questions
After you have gotten someone to begin your survey, you’ll want to avoid loosing them in mid air. Therefore, spend a little extra effort on sequencing and phrasing the first minute or two of the survey. Begin with a few easy closed (yes/no, multiple choice) questions so the subject can choose his comfortable pace. Your primary objective is to start out by demonstrating the legitimacy and intent of your survey.
4. Ordering Makes A (Big) Difference
The order in which items are offered, and the order in which alternatives within an item are offered, makes a (big) difference. You can suffer from both primacy and recency effects. Options that appear at the beginning of a (long) list typically have a disproportionately higher chance of getting chosen (“primacy”). Similar effects have been observed towards the end (“recency”).
To mitigate the order effect of answer alternatives you can produce multiple versions with different ordering of alternatives. This should balance out order effects, to the extent possible. You’re still stuck with undesirable variance (from alternative sequencing), but that’s certainly better than “blindly” accepting bias because of (uncontrolled) ordering effects.
5. Don’t Be Stingy With Space
The way you physically present response alternatives has a material impact on which option will get chosen. In particular web and paper questionnaires are prone to a disproportionate number of first or early alternatives being chosen.
To save space the use of drop down menus or list boxes in web surveys is customary. The reason is managing screen space. It has been demonstrated in research that drop down menus and list boxes worsen the problem of respondents going for (one of) the first options. A comprehensive display with (mutually exclusive) radio buttons gives the least biased results (though certainly not unbiased), so apply this when feasible given your number of alternatives.
6. Context Makes A Difference
The context in which people respond to questionnaires is known to influence the outcomes. “Context” can be taken very broadly to mean just about everything in the respondents’ geographic but also psychological or “temporal” environment. So what is going on “around” the subject, and what happened prior to answering the survey.
Research has demonstrated that people respond relative to illustrative images. If you’d ask: “do you consider yourself attractive?”, you will get more positive responses when you picture someone homely and unappealing rather than picturing Brad Pitt or Cameron Diaz (pick your own favorites…).
7. Question Phrasing Is Simplicity Poetry
Creating well-structured, simply written questions is fine language art. Careful crafting is required to ensure valid responses. Questions need to be clear and direct. Avoid negatives altogether if possible. Write concise and use simple language. You’re not testig respondents’ vocabulary. Choice alternatives should be unambiguous and comprehensive. In other words, every conceivable answer needs to be covered, and even so, n/a, etc., are often also required.
Ultimately, there is no surrogate for comprehensive testing of your survey instrument. However, in most instances, there simply isn’t enough time and money. But even with significant budget you will get further, and get there faster once you master this “art.” And an art it is, which is why I liken it to poetry. Feedback on your writing can and should come from (extensive) testing (see also tip# 8).
8. Test Your Survey Before Rolling Out
The best way to test a survey instrument is by comprehensive item analysis (an advanced quantitative method). However, this is pretty labor- and resource intensive. And often there isn’t enough time. Don’t let these challenges seduce you into neglecting tests altogether. Even a basic usability test (see also a previous newsletter on usability research) can quickly surface the most important problems. You may not have time nor resources to fix them all, anyway. Jacob Nielsen has coined the term “discount usability testing” (see also Steve Krug’s latest book Rocket Surgery Made Easy (2009)
to describe these low key methods that are simple yet effective, and that anybody can do.
One such method is called the “think aloud method” which aptly describes it: subjects are asked to think aloud while they’re filling out your survey. This will quickly surface egregious errors. It’s also very well suited to determine if all your language is meaningful in their terms. And whether the same term means the same thing to all subjects, for instance.
9. Rewards Improve Response
Surveys usually target samples (see also a previous newsletter on survey sampling). From this sample you want to maximize response. Better response increases coverage. Coverage improves validity. Higher response even more so (given a random sampling procedure).
Rewards have been shown (in research) to improve response. There is some trepidation towards direct financial rewards. Some feel this attracts the “wrong kind” of respondents (biased), although there is no conclusive evidence to support that hypothesis. In contrast, the fact is well established that rewards improve response, and that monetary rewards are highly appreciated. But alternatives are available. Being polite, saying “thank you” is rewarding, too. But you might also consider providing respondents with survey outcomes, results, actions taken as a result, etc. The general tenet is to make responding important and legitimate.
10. There Is More To (Research) Life Than Surveys
As useful as surveys are, there are many other methods for gathering research data. Because surveys are such an intuitive and obvious way forward, you might loose “inspiration” to think of alternatives.
Besides primary data collection techniques like a survey, you could also consider secondary sources. We know, for instance, that average breast sizes are growing and are highly variable across countries. We did not learn this from surveys or direct observation, though. These statistics are derived from bra sales figures, controlled by an array of covariates. Likewise you can always ask yourself what alternative methods besides surveying might be available to answer your research questions.
