Tom’s Ten Data Tips – December 2006
Market Research
Market research can deal with both people’s behavior as well as their attitudes or opinions. Research methods can be organized from subjective (e.g. self-report questionnaires) to objective (observation or behavioral data). This two-by-two matrix isolates subjective attitudinal research (e.g. satisfaction surveys), objective opinion research (e.g. mystery shopping), objective behavior research (e.g. point of sale transaction data), and subjective behavior research (e.g. self-reports).
The reflex often is: “we have a problem, let’s do research”. Within this two-by-two framework one can find an appropriate method that suits the initial question and problem requirements. Choosing the appropriate research approach depends heavily on framing the problem, but also on surfacing of assumptions. If done well, accurate problem analysis may also lead to the conviction that not doing research is the preferred option.
1. People Don’t Always Do What They Say
Most people equate market research with self-report questionnaires, which is neither true nor justified. In all fairness, this prejudice comes from a forceful “habit” of researchers to default to such research designs. This automatic connection has contributed to a somewhat flaky reputation of market research.
The function of market research is supporting marketing decisions. This is why reliable conclusions about customers’ behavior are the litmus test, and why sometimes the problem requirements can not be met in an acceptable time frame (or budget constraints). This allegedly ‘broader’ function of marketing support will often point to more diverse methodologies, or not doing research at all!
2. Subjects Display ‘Normal’ Behavior In Their ‘Natural’ Surroundings
When you take research subjects out of their “normal” habitat, and subsequently question them about behaviors in their “ordinary” lives, validity is at risk. If there is not enough time to do the appropriate research, then choosing a poor alternative is, well, hardly an alternative.
Research participants will answer almost any question, and comply with almost any request (as Milgram convincingly demonstrated in 1963: “Behavioral study of obedience”). However, this doesn’t guarantee truthfulness of responses, nor does it imply that subjects are liars. It just means the research design is improper.
3. Research Design Needs To Fit The Objectives
There are two axes along which to organize research approaches. The design can be:
- Geared at peoples behaviors versus their attitudes or opinions
- Based on objective versus subjective methods
Observed behavior is the objective for cameras reporting shopping behavior, customers moving through isles, or eye tracking devices.
Observed attitudes can come from field studies, or mystery shopping, etc.
Self-reports on behavior are the bread and butter for market researchers (voting polls, questionnaires on buying behavior, etc.).
Self reports on attitudes are seen, for instance, in satisfaction research.
It is adamant that the choice for a design be governed by the objective(s) for doing research and that it matches the research question.
4. Qualitative and Quantitative Research Each Have Unique Merits
The choice between a qualitative or quantitative research approach should never be based on budget. Both approaches have merits, and the appropriate choice depends solely on the objective of doing research.
If the research question deals with “what”, “how much/many”, quantitative research is in order. If the research question has to do with “why”, “motives”, or discovering what vocabulary is common in a prospect group (e.g. for questionnaire design or SEO), then qualitative research is in order.
5. Qualitative And Quantitative Research Have Their Own Reliability Measures
Reliability in qualitative research is achieved when “convergence” occurs. You observe this when new focus groups (or individual interviews) no longer provide additional insights with regards to the research question, when only points are being raised that earlier subjects already have brought up. The results are then “saturated”.
Reliability in quantitative research means that outcomes are specified within numeric boundaries. It is possible to derive these numbers within the research sample (if appropriately drawn). When calculated properly, these should (almost always) be presented as asymmetric confidence intervals.
6. Time Pressure Is Never An Excuse To Ease Up On Ethics
Ethical considerations are typically the first to ‘give way’ when a deadline for research is at risk. Careful communication with research participants, for instance, may take longer than seems affordable.
To succumb to such pressures is never a good idea. ‘Semi-ethical’ compromises have a nasty habit of coming back to bite you in the butt…
7. Internet Offers New Research Opportunities
The web isn’t just a less expensive medium to interact with research subjects, it offers new ways of interacting, too. Web panels are more economic because establishing contact with research ‘panel’ members is easier and faster. Additionally, this allows for shorter research turn around times. “Computer game-like” interfaces (e.g. virtual worlds like “secondlife”) allow for efficient yet real-life like research designs.
Unfortunately, most internet panels are still being used as if they were presented an old-fashioned paper-and-pencil questionnaire. The interactive and graphic possibilities that the web offers are not often taken advantage of.
8. Product Concepts Are Almost Impossible To Research
Regardless of what agencies will tell you, it is not possible to do valid research into product concepts. The reason behind this is twofold:
- Research subjects get instructions to imagine what they don’t know yet
- Even for product concepts where a “prototype” is available, the subjects have never become familiar with the products
The research may, however, be very reliable, accurate and reproducible. That still doesn’t mean it is valid.
Many concepts ‘failed’ in initial research (e.g. text messaging), yet went on to become a big hit in the marketplace. The reverse occurs too, unfortunately.
9. Most Internet Research Panels Aren’t Really Panels
Large research agencies are all offering access to their panels these days. In reality, very few of these so-called panels are true “panels”. To qualify as a research panel, some type of longitudinal research should be involved. A concern with these internet panels is that qualification and identification of research participants is flaky at best.
What are offered as “panels”, in reality are typically convenience pools of potential subjects who have expressed some interest in being eligible for research. If sufficient background variables are available some kind of stratifying can be done. Given the fickle nature of some of these ‘panels’, response rates are lower than might be expected from a true panel, and stratifying options are usually quite limited.
10. Solid Research Design Hinges On 3 Elements
When a research question is converted to a plan, there are 3 distinct elements involved:
- Conceptual design
- Technical design
- Research design
The conceptual design translates the research objective and problem definition into the specific question as it will be answered by research
The technical design refers to the choice of method, like survey, (quasi-) experiment, observation, etc.
The research design ensures that the project fits into the planning horizon to deliver results within the relevant timeframe.
Naturally, all three elements have to work in conjunction to deliver meaningful and compelling results. The research outcomes need to relevant to the client, and the chosen method should be both reliable and valid.
Further reading
Some excellent books on Market Research:
Robert P. Abelson (1995) Statistics as Principled Argument.
ISBN# 0805805281
ISBN# 0072821116
Gary Lilien & Arvind Rangaswamy (2004) Marketing Engineering.
ISBN# 1412022525