Tom's Ten Data Tips - September 2011

Writing Survey Questions

Writing (good) survey questions may seem more of an art than science. But there are many guidelines on using language properly. And common survey pitfalls can be avoided, too. In many ways, writing good survey questions is like gathering requirements: by focusing on what you are trying to measure with the survey, you attempt to eliminate as much ambiguity as you possibly can.

There’s a plethora of guidelines that have emerged from the last few decades since the use of surveys has boomed. Numbers are for self-reported “measurement” of “real” properties like income, age, duration, cost, etc. Likert scales and semantic differentials are for measuring feelings. Avoid asking double-barreled questions (asking two questions at once), no technical jargon or acronyms. Use different fonts/colors for instruction versus the “actual” questions, etc. For every question, you should wonder: will every respondent be able to answer this? In the same way?

1. Personalize Your Questionnaire Invitation

Solid empirical research has shown that a personalized invitation to participate in a survey tends to increase response rates. People also appreciate their privacy, so if you are going to use this tactic, make sure you display how you got access to their contact details. This works best if the invitation to participate comes from a familiar and trusted source. To ensure familiarity it can make sense to precede the actual questionnaire with an announcement that includes the possibility to opt out. This manages aggravation, and also helps to build trust.

2. Use Specific Wording

When you write survey questions, your objective is to have all respondents interpret the question in the same way. That’s a lot to ask for, but needless to say you at least want to reduce variance as much as possible. Therefore, avoid using words like often, generally, typically, usually. Instead specify the frequency, amount, etc. Emphasize crucial words in the text for clarity using italic or bold typeface.

You can not be too specific! But you can be ambiguous. It’s amazing when we do quantitative research how much room for different interpretation people find – even with seemingly unambiguous phrasing! Be as specific as you can think of.

3. Avoid Open Ended Questions If You Can

If at all possible, make all questions either a rating or closed form (multiple choice). Open ended questions tend to cause more problems than they solve. It is non-obvious how to analyze and/or report findings. Categorizing or clustering responses happens only after the fact, and is subject to interpretation. Open-ended questions are (much) more time consuming for the subject, and therefore get considerably higher skip rates.

The one exception that we’d be willing to cede is a leftover “other” response category, offering an opportunity for respondents to specify. But even then it is more of a courtesy than anything else because reporting on that free format is subject to the same problems. Open-ended questions are near impossible to analyze. It’s more like gathering “data” for a brainstorm. Is that what your survey is for?

4. Cut Sentences Up

The simplest and easiest way to phrase isn’t always shortest. For those who took it: remember Latin class? These texts were incredibly dense and therefore so hard to decipher (at least for a mediocre student like me…). All redundancy in Latin is removed, and transferred to cases.

For readability you want to cut up your message in multiple short sentences. This will introduce redundancy which is good. Texts become more readable. Confusion gets eliminated. Reading speed will easily offset extra length, and at the same time drive out ambiguity.

5. Make Sentences “Flow”

Material that you can read very fast is typically easy to comprehend. So keep sentences brief, you’re better to err on the staccato side than on the long side. Short is sweet. For enjoyable reading, you’ll want to mingle short and longish sentences. Provide specific examples rather than abstract descriptions. Metaphors, stories, examples, use anything to make your prose more lively. Write as if you’re having a one-on-one conversation with the respondent.

6. Cut Down On Fluff

After you’re done writing the entire survey, review everything you have produced to determine if it is essential. Important! Do not review before you’re done. Do not censor until you have completed the draft. Many people are too critical of their own work. You don’t want to tinker with material that is 95% good.

Go over the survey, and question every word or phrase for necessity. Can you find more compact, alternative words to replace parts of sentences? Which adjectives can you do without? You’re looking for the shortest formulation without loosing intent. After one entire pass, return the next day and repeat. You’ll find there is even more you can scrap. Do so. Nobody ever complains a survey was too short.

7. Avoid Leading Question Bias

A major problem with “purposeful” questionnaires is that the very act of raising attention for something specific primes the respondent and causes undesirable bias. A perfect (bad!) example we recently encountered went like this: “We have recently upgraded our website, what do you think of the new design?”

First of all, the website was changed; it is up to the users to qualify this as an upgrade or not. Unless the change works appallingly bad, phrasing the question like this will tend to lead to more favorable responses compared to “neutral” wording. Secondly, by drawing attention to the change in this way, you have raised awareness. How many subjects had noticed a change without priming them? We’ll never know.

8. Measuring Income Is Tricky

For a number of reasons asking for people’s income is a challenge. In many cultures, asking about salary is sensitive (money in general, actually). But apart from the social and psychological aspects, there are genuine measurement issues, too. How to define income? Few people precisely know their yearly income (by heart). Should that be gross or net? And what do we mean exactly by “net”? Which taxes or contributions have been deducted? The problems with an exact definition are innumerous.

People generally tend to be more aware of their exact monthly or weekly income, more specifically, what they receive in their account. But here the problem (again) is to accurately define “net income.” On top of that, for part-time employees (a large proportion, certainly in Europe) it proves difficult to elicit their working hours, let alone factor in tax regulations to calculate equivalent monthly (or hourly) pay, etc. Fringe benefits further complicate this. In short, measuring income precisely is a nightmare!

9. Be (Very) Specific About Education Questions

Education level is another notoriously difficult variable to measure. At least to measure it right. When we refer to “highest education level attained”, does that include unfinished education? And what about education you are currently attending? What if you switch between levels? Have you “finished” when you’re done but haven’t received your diploma, yet? Etc. Ambiguity abounds.

Another reason why education is such a hard categorical variables to measure is because there are huge international differences (ambiguity) in naming education levels. So if your survey covers multiple countries, you will need to “standardize” these differences in some way. Since you will most likely want to create categories anyway, it makes sense to think about this beforehand (see also tip# 3).

10. The Proof Of The Pudding Is In The Eating

After you are done writing the draft, you edit it down for brevity and clarity (see also tip# 6). Now you’re ready for your first test pilot. Ask two or three representative subjects to go through the entire survey while thinking out loud. You can certainly use more, but often you bump into a few so-called “head slappers” (aka: “Why didn’t I think of that before??”) that you’ll want to fix immediately. More subjects then simply means more evidence of your oversight, not necessarily better usability research (see a previous newsletter on that topic).

Test for clarity, flow, and proper understanding of all language. Does the same question mean the same thing to all test pilot subjects? Try to use words that are familiar to all respondents, and check whether this is the case. Often your initial pilot will surface enough issues to merit a rewrite. Then rinse and repeat. After a successful “real life” test on a small group you can gather empirical data how long it takes to complete the questionnaire, etc. Make sure enough time is planned in between iterations to accommodate all findings and necessary changes. Now roll out.

Further reading

Some excellent books on Writing Survey Questions:

Business Writing and Communication.
Kenneth Davis (2005)
ISBN# 0071441271