Tom’s Ten Data Tips – December 2010

Internet Surveys

Internet surveys offer significant benefits over paper and pencil in terms of lay-out, formatting, and most importantly routing of questions. You have maximum control and liberty to use colors, logos, pictures, etc., and to manipulate their usage in light of your research objectives. They also enable fast turnaround times for analysis and reporting.

Because respondents can’t gloss over a paper form, the web interface needs to supplant that option. You will probably want to use either fast forward and backward arrow buttons (do the keyboard arrows work??) and/or a bar which keeps track of what percentage of the survey is done at any point in time. Respondents like to feel “in control.”

1. Usability Rules!

Because of the “fickle” nature of internet visits, abandoning the survey is literally one mouse click away. Therefore, take extreme care to avoid any usability issues, and to ensure respondents are sufficiently motivated. If you promise the survey will only take five minutes, you can’t make it last ten, or people will abandon the survey. Make it painless. And make it quick.

Typically, respondents like to be able to navigate backwards and forward through the survey. Is this “technically” feasible? Will it enforce reminders of unanswered questions? It’s helpful to have a “resume button” jumping to the first unanswered question.

2. “Lead” Respondents Into Your Survey With “Easy” Questions

After you have gotten someone to begin your survey, you’ll want to avoid loosing them in mid air. Therefore, spend a little extra effort on sequencing and phrasing the first minute or two of the survey. Begin with a few easy closed (yes/no, multiple choice) questions so the subject can choose his comfortable pace. Your primary objective is to start out by demonstrating the legitimacy and intent of your survey.

3. Only Underline Hyperlinks

The question often arises how formatting of text can support structure within and across survey items. It makes sense, for instance, to use different colors and/or different fonts/typeface to distinguish between instructions aimed at the respondent and the actual questions to be answered. Be consistent throughout the survey, and preferably comply with corporate style guidelines.

But whatever you do, never, ever underline text to emphasize it. One of the few universal design guidelines across the web is to reserve underlined text for hyperlinks. In every usability study we have ever done where this guideline was ignored, we had to observe frustrated users repeatedly clicking on “hyperlinks” that failed to take them to the next page. Please don’t repeat that error.

4. People (Only) Respond If They Trust The Researcher

Empirical research shows that if the sender of an invitation for research (often a survey) is known to the respondent and trusted, their chance of responding goes up dramatically. What this implies is that if you decide to use an agency (usually a good idea), then you’d still prefer to send out the invitations yourself. Or else have the agency send out the research invitations using your email account as the sender. A second best would be to send out the announcement yourself, and then have the agency follow up within a week afterwards with the actual link to a survey form.

5. Randomize Alternatives To Mitigate Order Effects

When subjects are faced with a (long) list of response alternatives, the first (“primacy”) and last (“recency”) options are most likely to get chosen. When feasible given the alternatives, the best way to deal with this problem (bias) is by randomizing the order in which they are presented. What this means is that for every subject, the list of choice options gets shuffled. This way, every alternative has an equal chance of coming first or last. This is one of the powerful ways in which computer aided interviewing is superior over paper and pencil surveys (see also tip# 5).

6. Use Skip Logic Whenever Possible

By using “filter-”, “conditional questions”, or “skip logic” questions, you assure that subjects only get questions that are relevant to them. In paper and pencil surveys, there are always respondents who get the skip logic wrong. When you program your survey well, skip logic should not cause any problems on the web.

The fact that you only need to present subjects with questions that are relevant to them is a big plus for internet surveys. For some designs like conjoint analysis, this opens up possibilities that traditional methods lack: by looping through a vast pool of pairwise comparisons, you can handle research questions that otherwise would strain respondents too much. If you manage to build in the required intelligence for adaptive conjoint analysis (ACA), research designs can become feasible that would otherwise be either too cumbersome or costly. At a minimum it will help to shrink confidence intervals.

7. Let Subjects “Rank” Ranking Questions

Unfortunately, it is still common to ask respondents to fill out numbers like 1-5 next to alternatives they were asked to rank. This has three significant drawbacks. First of all it is time consuming, and therefore painful for the respondent. Secondly, it is less intuitive, and therefore less valid. People simply don’t rank using numbers, they rank according to order. And they might even confuse (despite your clear instructions…) whether one or five comes first. Thirdly, it makes it more difficult for people to change their mind while they’re working on the task. Therefore they’ll be less motivated to “play” with the ordering.

The “right” way to do this is to present the options graphically where point and drag moves the alternatives in vertical order. Some interfaces use arrows pointing up and down to jump or drop places. That has been shown to work, too. Most importantly, ensure that it works properly, easily, and intuitively.

8. Give Tangible Rewards To Subjects

Research shows that giving rewards to subjects “helps” to motivate them to participate and complete the survey. This same research shows that specific, “sure” and tangible benefits motivate subjects the most. So it’s better to give away $10,= than to offer a 10% chance of winning $100,= for instance, although economists will point out that the expected value of the offers is identical. Apparently consumers don’t buy into the assumptions of rationality that economists propose (gee, who would have thought that?).

One of the easy and straightforward offers you can make your respondents is a report, or at least insight in what was discovered from the survey. This doesn’t cost much extra, and appeals to intrinsic motivations of people for your research objectives.

9. If You’re Going To Contact The Respondent, Make It Painless

Sometimes for reasons of customer service (when respondents voice complaints during the survey), you might want to contact respondents afterwards. Or at least offer to contact them. Of course you need to make good on that promise… Another fine reason to contact respondents might be because you promised them to send a report or summary of results (see also Tip# 7).

If and when you decide to do this, make sure that capturing the respondent’s contact details is quick and painless. Follow best practices for “snail mail” forms, reuse email details whenever possible in your forms. Few things look sillier than contacting someone by email, and then asking him to fill out his email address again…

10. Make The Web “Shine”

Graphics and design are important for paper surveys, of course, but when you use the full potential of HTML (or XML) you can take look and feel to a new level in an internet survey. Interactivity, intelligent routing, and flexible question formats are unparalleled. You can offer context sensitive help functions that can be available everywhere, at any time. For maximum usability, the content of the help function can be made to adjust to the specific location within the survey where it is called. This is where good design can really shine.

As an additional bonus for internet surveys, the speed of sending and response is a big win. If you set your research questions up well, turnaround times of days are possible. For a similar survey, the turnaround times would be (at least) weeks in paper and pencil format. And nobody minds that you can skip costly and error prone data entry ☺

Further reading

Some excellent papers on Internet Surveys:

Comparative Response to a Survey Executed by Post, e-mail, and web-form. Journal of Computer Mediated Communications 6(2)
Yun & Trumboo (2000)

Using email questionnaires for research: Good practice in tackling non-response. Journal of Targeting, Measurement and Analysis for Marketing 14(4)
Michaelidou & Dibb (2006)