November 3, 2005 | Issue 45

Editor's Note

	MR Perspectives is a twice-monthly newsletter that provides perspectives on market research topics of interest, best practices tips, emerging trends, quick case studies, and other useful information. Subscribe View Past Issues

Events

	Nov. 6-10, 2005 San Francisco The Marketing Research Event The Institute for International Research is sponsoring The Marketing Research Event at The Sheraton Palace in San Francisco, California.

	Nov. 6-11, 2005 Atlanta Advanced School of Marketing Research The American Marketing Association is sponsoring Advanced School of Marketing Research at the Atlanta Executive Education Center for the University of Georgia Terry College of Business in Atlanta, Georgia. Nov. 13-16, 2005 Chicago Symposium for the Marketing of Higher Education The American Marketing Association is sponsoring a Symposium for the Marketing of Higher Education at the Downtown Marriott Magnificent Mile in Chicago, Illinois.

More Events

	Nov. 30-Dec. 1, 2005 Covington, KY Data Collection Conference The Council of American Survey Research Organizations (CASRO) is sponsoring a Data Collection Conference at the Marriott Cincinnati at RiverCenter in Covington, Kentucky.

Don’t Skip The Pretest

When schedules surrounding a research project are demanding, a common suggestion is to cut short or even do away with the pretest, in which a relatively small number of respondents, typically between 25 and 75, are surveyed. But the pretest serves several purposes which warrant paying it due attention, even when the time frame is tight.

From the standpoint of the client, the pretest offers an opportunity to improve the survey instrument. This is especially true in the case of phone surveys, where clients can listen to phone calls. Monitoring the pretest offers opportunities to:

Identify potential problems with the survey. These are brought to light when respondents get confused by a particular terminology or phrasing of a question. Sometimes, interviewers experience problems such as needing to repeat questions and needing to correct misinterpretations of questions.

Identify additional opportunities to collect feedback regarding different aspects of the survey topic that had not yet been considered. This often occurs in sections of the survey where the respondents want to say more or volunteer other information during the interview.

The pretest also allows researchers to avoid costly miscues by identifying any programming or other related data collection issues that might otherwise only be identified upon completion of the study. Any potential issues either in survey design, programming, or data collection are much more manageable if identified after 30 completed surveys as opposed to 300.

From a statistical standpoint, no matter how talented the researcher, the pretest will often bring to light potential sources of bias and error in the survey design.

The pretest also can be used to narrow a larger list of attributes regarding a desired topic to a much smaller set, dropping weaker items off the list before the survey is fielded to the larger sample.

Other purposes of the pretest are to see if respondent interest is aroused by the survey, if respondent attention can be maintained and if the survey has a natural flow.

When all is said and done, if the resulting survey changes are deemed substantial enough, often a second pretest is warranted in order to do some last polishing and trimming of the survey instrument.

Tips For A Successful Pretest

Below are a few tips to help make your next pretest as successful as possible. While certainly not exhaustive in nature, the following list provides some good starting ground.

Let participants know they are taking a pretest. Respondents can help to refine the survey instrument when they understand that is what they’re being asked to do.

Upon survey completion, ask respondents about the flow of the questionnaire and whether it was difficult to complete or boring. Also ask about specific questions, whether they found them confusing or how they interpreted their meaning.

Try to conduct one pretest as much like the final survey will be administered as possible. If the final questionnaire will be mailed to respondents, take the time to mail the survey out for the last pretest. Doing so will make sure that all elements of the survey administration are being pretested, and will remove any variables that would have the results of your final sampling differ from your pretests.
On that note, it is also worthwhile to have, in at least one of your pretests, the sample closely resemble your final sample as possible, even if that means using some of the actual sample.
Take a look at frequencies when the pretest is finished. If there is little to no variance on a particular item, it may not be measuring the intended concept as well as you would like.
Look for too many “Don’t Know” responses as a sign of a poorly constructed scale.
Last but not least, time the survey. If the survey is going to take respondents too long, they likely won’t finish it.