Qualtrics survey software from Utah-based Qualtrics is an excellent web-based survey software package that offers a fantastic array of question types, a well-designed survey development interface, good fielding/survey promotion capabilities and a powerful reporting engine. It has both panal management features and multi-users capabilities and should definitely be a contender if you’re a corporate research department or academic organization looking for a survey system.  (more…)

Julie Lemaster, an MBA student at the University of California- Riverside, has written an interesting paper (which is posted to the Sloan Center for Internet Retailing web site) entitled "Online Panel Management and Strategies: An Introduction for Managers." It is an introductory guide to managers who have been asked or have decided it is time to start looking into online market research for their companies.

Lemaster contrasts full-service providers, such as M/A/R/C and SSI against complex and potentially expensive "self-serve" packages from Confirmit, Globalpark, GMI, and SurveyZ to low cost providers such as QuestionPro, Survey.com, SurveyMoney, and Zoomerang.

The abstract of the paper summarizes it as providing…

"…an introductory guide to managers who have been asked or have decided it is time to start looking into online market research for their company.  The size of the company you work for does not matter, as we will discuss several methods that can be used for any size company or investment level.  This paper is for managers who want to quickly learn the basic issues of online market research panels.  It will also be useful to managers who need to become familiar with some of the major providers of online panel management."

In addition to reviewing the providers and the various strategies for organizing your panel, Lemaster discusses a variety incentives that can be used to motivate and retain participants, such as lotteries, bonus points, and raffles.

• Online Panel Management and Strategies: An Introduction for Manager

Unless you have a practically limitless number of potential survey respondents at your disposal, you probably spend a lot of time thinking about how you can maximize your response rates — that is, the percentage of people that you invite to take your survey who actually take it. Several of the professors at the UCR Sloan Center for Internet Retailing have put some time and effort into researching the problem and have recently published a working paper in which they discuss their findings to date.

The paper is entitled "An Optimal Contact Model for Maximizing Online Panel Response Rates," was written by Scott A. Nelsin, Thomas P. Novak, Kenneth R. Baker, and Donna L. Hoffman. Here are a few of the points I found interesting in the background section of the paper:

• Response rates from online surveys tend to be lower than from traditional survey methods. Some of the theories that have been put forth to explain this include respondent privacy concerns, technical problems, respondent confusion, and poor design.
• Standard methods used to increase response rates in the real world may not work online.
• Nearly 80% of consumer goods and 74% of B2B companies use online panels.
• Newly formed online panels often experience high response, although these levels drop quickly without proper management, which includes attractive incentives, pruning of non-responders, recruitment of new panelists, personalization of messages, and creating a sense of "community" among panelists.
• Online panels are very inexpensive: telephone surveys cost anywhere from $15 to $20 per respondent; mall intercepts cost around $10 per complete, while online surveys tend to cost $1 to $2 per response (for the panel owner).

The "meat" of the paper discusses a predictive model for categorizing respondents into different classes based on their expected response rate and then using those classifications develop an optimization model for determining how many invitations to send to each group in order to maximize a broad-based response.

The abstract of the paper:

We develop and test an optimization model for maximizing response rates for online marketing research survey panels.  The model consists of:  (1) a decision tree predictive model that classifies panelists into “states” and forecasts the response rate for panelists in each state, and (2) a linear program that derives a plan specifying how many panelists should be solicited from each state in order to maximize response rates.  The linear program is forward looking in that it optimizes over a finite horizon during which S studies are to be fielded.  It takes into account the desired number of responses for each study, the likely migration pattern of panelists between states as they are invited and respond or don’t respond, as well as demographic requirements.  The model is implemented using a rolling horizon, whereby the optimal solution for S successive studies is derived and implemented for the first study; then, as results are observed, an optimal solution is derived for the next S studies, and the solution is implemented for the first of these studies, etc.  The procedure is field tested and shown to increase response rates significantly, compared to random selection and the heuristic currently being used by panel management.  Implications are discussed for further model development, implementation issues for online panel managers, and for the broader area of optimal contact models in customer relationship management. 

Why you wouldn’t just send all of your invitations to the most responsive group? Well, if you send out surveys infrequently and if you feel confident that you high responders are representative of your entire market then there is not reason not to do that. However, if you do send out a lot of surveys and you want to capture the broadest sample, then you need to find a way to maximize your entire panel without relying too heavily on a few dedicated panelists. That is where this model can come in handly (if you can figure out how to implement it, of course).

• Optimal Contact Model for Maximizing Online Panel Response Rates" at UCR eLab and Sloan Center

Research provider Knowledge Networks (Menlo Park, CA) has published an interesting nine page white paper entitled The Decisions Maker’s Guide to Online Research. Although the document is clearly geared towards illustrating how the Knowledge Networks panel methodology is better than others, the pamphlet does provide a thoughtful framework for deciding between web-based surveys with self-selected panels, web-based surveys with probability-based internet panels, mall intercepts, and mail surveys.

It identifies two types of online panels:

• Self selecting panels, which anyone can join — a "convenience sample" of the internet, and one that is likely to contain "professional respondents" and possibly even competitors trying to get insights into your secret plans;
• Probability-based Internet panels, which are (painstakingly) built by randomly selecting people, calling them, and then inviting them to join the survey panel.

One of the primary sources of data for the Knowledge Networks publication was a study conducted by the Stanford Institute for the Quantitative Study of Society entitled "Comparing the Results of Probability and Non-Probability Sample Surveys." The study begins by acknowledging that in general, the same survey conducted by two firms with the same methodology will usually yield comparative findings. However, most studies that led to this conclusion focused on surveys conducted in the same mode with comparable sampling methods. The folks at Stanford wanted to find out what would happen if the mode and sampling methods differed.

Nine data collection firms participated in the study: seven of whom use a self-selecting, volunteer sample (self selecting panel) and the other two who used probability-based panels (one that used a probability based telephone sample, the other (Knowledge Networks) that used an Internet-based probability panel). Each data collection firm asked their respondents the same set of questions, and the results were compared against benchmark probability-based responses.

The findings of the folks at Stanford were that the results were "remarkably comparable" across the board. Knowledge Networks had the most accurate findings, followinged by SRBI (telephone survey) and Harris Interactive (volunteer sample) who tied for second place (all of the others were about equally as accurate).

A few questions led to bigger differences between the self-selecting and the probability based methods: for example, volunteer respondents tended to be more comfortable using computers than probability-based respondents. Otherwise, however, it would appear based on the results of the study that a volunteer sample base will ultimately lead to results that are closely comparable to the more expensive probability-based sample.

Read the Knowledge Networks publication.
Read the press release announcing the publication.
Read the results of the 2005 Stanford study.

ResearchExec, owned and operated by the Fairfield, Connecticut based company of the same name, is an entirely web-based survey system that is available either as a hosted solution or as a package you can install on your own server. It offers a tremendous amount of customizability as well as a number of advanced features such as the ability to set quotas for individual questions and advanced survey logic. It has integrated panel management that allows you to send surveys to specific members based on their responses to previous surveys. The survey development system, while elegantly designed and extremely flexible, is tedious to use and does not play well with FireFox 1.5 or Internet Explorer 7 RC1. It offers very little in terms of a reporting system, and you should expect to do most of your analysis in a separate program such as Excel or SPSS. Because of the focus on advanced features, the sharper learning curve and the lack of a reporting engine, ResearchExec is more likely to appeal to users at a professional research firm and not those looking for a quick easy way to produce and report on internet research.

(more…)