Research provider Knowledge Networks (Menlo Park, CA) has published an interesting nine page white paper entitled The Decisions Maker’s Guide to Online Research. Although the document is clearly geared towards illustrating how the Knowledge Networks panel methodology is better than others, the pamphlet does provide a thoughtful framework for deciding between web-based surveys with self-selected panels, web-based surveys with probability-based internet panels, mall intercepts, and mail surveys.

It identifies two types of online panels:

• Self selecting panels, which anyone can join — a "convenience sample" of the internet, and one that is likely to contain "professional respondents" and possibly even competitors trying to get insights into your secret plans;
• Probability-based Internet panels, which are (painstakingly) built by randomly selecting people, calling them, and then inviting them to join the survey panel.

One of the primary sources of data for the Knowledge Networks publication was a study conducted by the Stanford Institute for the Quantitative Study of Society entitled "Comparing the Results of Probability and Non-Probability Sample Surveys." The study begins by acknowledging that in general, the same survey conducted by two firms with the same methodology will usually yield comparative findings. However, most studies that led to this conclusion focused on surveys conducted in the same mode with comparable sampling methods. The folks at Stanford wanted to find out what would happen if the mode and sampling methods differed.

Nine data collection firms participated in the study: seven of whom use a self-selecting, volunteer sample (self selecting panel) and the other two who used probability-based panels (one that used a probability based telephone sample, the other (Knowledge Networks) that used an Internet-based probability panel). Each data collection firm asked their respondents the same set of questions, and the results were compared against benchmark probability-based responses.

The findings of the folks at Stanford were that the results were "remarkably comparable" across the board. Knowledge Networks had the most accurate findings, followinged by SRBI (telephone survey) and Harris Interactive (volunteer sample) who tied for second place (all of the others were about equally as accurate).

A few questions led to bigger differences between the self-selecting and the probability based methods: for example, volunteer respondents tended to be more comfortable using computers than probability-based respondents. Otherwise, however, it would appear based on the results of the study that a volunteer sample base will ultimately lead to results that are closely comparable to the more expensive probability-based sample.

Read the Knowledge Networks publication.
Read the press release announcing the publication.
Read the results of the 2005 Stanford study.

Ray Poytner of The Future Place Blog recently posted an article entitled "The Push/Push Dichotomy – or why most reportals fail." It considers the value of "research portals" — which, by the way, refers to web sites set up by research firms so that clients can, on their own, download and review the results of ongoing or recently completed surveys, usually before the research firm goes in and adds their insight/analysis to the data. The question is whether or not setting up such systems is as effective as research vendors send the results to clients directly.

My own experience is consistent with Ray’s — when I have in the past set up research portals at my company — or even uploaded the research to a central web site or other location where the user has to go and download it for themselves — I think the result has generally been that the research is ignored or receives spotty dissemination. With all of the work everyone has to do, very few people really have the time to veer away from their day-to-day activity to download — or even worse, generate — marketing research reports.

That is, unless they’re looking for something in particular. Ray makes a good point that there are certain types of research that do lend themselves to reportals. The key, then, is to know when a reportal is appropriate — and when it will really be used — and when it makes more sense to disseminate results using more active (rather than passive) methods.

Read Ray’s article at the Future Place Blog.

SurveyMania.com certainly isn’t the first web site to collect information about different companies that pay/reward people for taking surveys, but their recent press release and their relatively full-feature web site make really makes you stop and think about online survey panels and whether or not it is the same set of people who join all of them.

The site presently provides instructions for joining over 30 different panels, broken into categories such as "Consumers," "International," "Technies," "Parents," and "Specialty."

The site makes money from selling ad space to survey panels and through banner ads.

In addition to maintaining the list of online research panels you can join, the site also promises to e-mail members (at no cost to join) new opportunities as they arise. The site claims to have over 750,000 members (only 65,000 of which are in the United States).

Read the SurveyMania.com press release at PRWeb.
Visit SurveyMania.com

SPSS recently announced the release of SPSS Dimensions 4.0, the latest incarnation of its enterprise survey and analysis suite that does everything from helping you create surveys to analyzing the data to generating reports. Before looking into the new features introduced in version 4.0, I thought it might first be interesting to explore the basic features of the system. In other words, what is SPSS Dimensions?

SPSS Dimensions isn’t so much an individual software package as much as it is a platform of several independent software packages that are able to work together in a relatively seamless fashion. Sort of like how each of the programs within Microsoft Office (Word, Excel, Powerpoint, Access, etc) can work independently (and be purchased independently) but also work very well together. Like Office, most of the packages that work with Dimensions are published by SPSS — although the platform has been designed to accommodate the integration of software written by 3rd party developers (and several such packages do exist).

Some of the programs that work with Dimensions include:

• SPSS mrDialer – Automated dialing for phone surveys
• SPSS mrInterview – Create and execute online surveys
• SPSS mrInterview CATI – Create and execute phone surveys
• SPSS mrPaper – Create and execute paper surveys
• SPSS mrScan – Scan paper surveys
• SPSS mrStudio – Manage and manipulate data
• SPSS Desktop Reporter – Create tables from local data
• SPSS mrTables – Interact with tables on your desktop
• SPSS mrTranslate – Manage translations of surveys and reports
• Techneos Entryware – Collect data using handheld devices
• SPSS Base – Analyze data
• Clementine – Data mining
• SPSS Text Analysis for Surveys – text analysis & categorization

At the core of SPSS Dimensions is the Dimensions Data Model, a set of components (openly documented and supported) which allow for accessing information about questionnaires and respondent data. It also deals with keeping track of changes to the questionnaire (versioning), translating both questions and data from one format to another, and managing data stored in multiple formats and platforms.

Visual representation of the Dimensions Data Model
(from the SPSS presentation "Using the SPSS MR Data Model")

The table above describes the role of the data model well. Data can be collected from multiple sources. Instead of each collection program storing the data in its own database, it instead sends it to the Dimensions Data Model which puts it into its own special format. When another program, such as a data processing program or a data analysis program needs the data it requests it from the Dimensions Data Model using standardized request formats (that just about any program can use).

Consider a project in which you need to collect data using three different survey techniques including a phone survey, a web-based survey and a paper survey. Even though you’re going to ask the same basic questions in each survey, you are still going to have to develop three completely different questionnaires in order to compliment each of the mediums, which further means you’re going to have to program the survey three different times (perhaps four, if you consider that you may be using scanning software to read some of your paper surveys). 

Even though the question is the same, it needs to appear different
across modes and across functions (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

After you’ve finished collecting the data (using three separate data collection tools, all of which store the data in their own, separate proprietary format that exports into the frustratingly simple CSV format, you’ll then have to combine all of the data into one file which you’ll then need to clean and prepare for analysis. Following analysis, you’ll export your results into yet another program.

Using the SPSS Dimensions Suite (or more specifically, software that is integrated into the Dimensions Suite) makes the process go much faster by optimizing the mechanics of designing and fielding your questionnaire and analyzing and reporting on the data.

Dimensions reduces the time it takes to conduct a complex research project
(Source: SPSS presentation "Discover it with Dimensions")

SPSS Dimensions has been developed based on the notion of "Design Once, use Many" so once you have created your initial questionnaire (either using a simple, graphic user interface found in mrInterview or the more advanced script driven interface provided by mrStudio — either package will allow you to import the text of your survey from MS Word), you can then quickly (and easily) set it up to deploy using multiple modes (paper, web, CATI, etc).

Perhaps one of the most exciting features of SPSS Dimensions is its multi mode deployment capabilities. Most surveys today require some amount of programming to deal with skipping, piping, the incorporation of outside data, and other advanced options. Ordinarily, each mode would require its own programming. SPSS Dimensions is designed so that you only need to write the script once and it will work the same in each context.

SPSS Dimensions allows you to program your survey once
and have it work on multiple platforms. (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

Dimensions not only helps you design and execute your survey, it also manages security, translating the survey into multiple languages, and manages multiple versions of your survey as well.

External databases containing participant details can be added at any time, and it can be used both in the survey and during analysis. Data from outside sources can be reviewed during the survey to check for inaccuracies, and it can even be updated based on responses given in the survey.

All of the data that is collected, regardless of how it is collected, goes into one SPSS Dimensions database where it can then be analyzed and reported on. Although Dimensions is an open platform that will allow analysis to be conducted in any program (it will export data into a variety of formats for other programs to use), the suite is optimized to work with several SPSS-published programs, such as mrStudio, mrTables, SPSS for Windows and Clementine. Results can then be automatically turned into interactive web-based reports or analyzed using Excel, Word or PowerPoint. Dimensions integrates all of the major capabilities provided by SPSS’s various data analysis packages, including SPSS Base for statistical analysis; Clementine for data mining, and SPSS Text Analysis for Surveys for text analysis and categorization as well as a variety of SPSS and 3rd party data collection and reporting tools.

Reasons to Consider SPSS Dimensions

• Powerful interviewing engine
• Open architecture
• Web-based user interface
• Easy to create surveys
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Easy to program (similar to VB Script)
• Common data storage format/interface
• Translation capabilities
• Faster development and analysis time
• Works with (some) third party applications
• Scriptable (write your own scripts to work with data)
• Integrates well with SPSS
• Integrates well with Excel

Reasons not to use SPSS Dimensions

• Expensive
• Limited to Dimensions compatible tools
• Complicated to set up and integrate with existing systems
• Requires lots of IT support

Learn more at the SPSS web site.

About a year ago Tim Macer gave a presentation at the SPSS Directions Conference entitled “Weaving, not drowning: An update on take-up and best practices in Mixed- and Multi- mode research.” Long, perhaps even academic sounding title, but actually extremely relevant to folks trying to figure out how to conduct and combine multiple modes of research (phone, web, paper, etc). In his presentation, he agenda covered the following questions:

• Who is doing it, how common is it?
• Why are they doing it?
• Why are some other people not doing it?
• Which modes work best together?
• When does it make sense to switch modes?
• What impact does it have on the data?
• What are the technical requirements?

One really neat concept I hadn’t thought much about was the idea of having a respondent start the survey using one mode (perhaps paper or the phone) and then have them finish the survey in another mode (usually the web). This has been found to help out when it is hard to retain respondents in one mode using a particular data collection method (perhaps they don’t want to hang out in your store for 20 minutes, or maybe they just want to get off the phone).

Multi-mode data collection will become especially useful as we adopt more mobile survey solutions — perhaps have the user start with a WAP based survey and finish up with a web based survey when they get home.

Read Tim’s full article at Meaning (PowerPoint).

Danielle Rodgers of Small Business Branding posted an interesting and thoughtful commentary about the potential benefits and pitfalls of using marketing research to make decisions in the new product development process. Her observations, which she follows up on with detailed explanations and suggested countermoves (so you should read her article) are as follows:

• Very few people will really understand the vision of your product when you describe it to them.
• If you ask people to find a problem they will (even if they never would have noticed it in real life)
• Talk is cheap — people don’t always (often? ever?) do what they say are are going to do (such as buy your product).
• Sometimes respondents don’t answer for themselves, but instead try to guess what the market is thinking.
• People don’t always have words to describe what they really want or need.
• Some people may have strong negative reactions to specific elements of your product.
• Just because one type of person doesn’t like your product doesn’t mean that others won’t.

Read Danielle’s full article at Small Business Branding.

Atlanta-based Mobile Transit Authority (MTA) and New York-basedscenarioDNA have launched a new mobile opinion solution called mDive, a turnkey mobile-based market research tool that engages consumers on the cell phones. The service works by using MTA’s mobile research platform to survey users through SMS text messages, WAP-based mobile internet connections, or a click-to-call voice-driven survey. Following collection, analysis, interpretation, reporting and recommendations are provided by scenarioDNA.

A complete mDive survey with 400 responses costs $6,500 and includes question development, delivery and management; an invitation in text format with link to WAP survey; a 10-question multiple choice survey; a CSV file of the raw survey data; setup, hosting and deployment of survey questions; and analysis of data and strategic recommendations in a PowerPoint presentation.

Read the the mDive data sheet (PDF).
Read the press release announcing the service at PRWeb.
Visit Mobile Transit Authority online.
Visit scenarioDNA online.

In my experience, the use of incentives, whether it is the opportunity to win a prize in a drawing/sweepstakes or points towards some kind of special reward, can make a big difference in the response rate to an online survey. But what to offer? In what context? Perseus is offering a free webcast on Wednesday, October 25 or October 31 at 1pm entitled "Learn the Secrets of Affordable Incentive Strategies" to help answer these questions.

The hour long presentation, led by Larry Nichter (EVP of Restaurant.com) and Brian Koma (VP of Services for Websurveyor) will include:

• Incentive best practices
• When to use incentives
• What types of incentives are most effective for increasing response rates, maximizing completion and improving data quality
• How to put incentive strategies in place that don’t blow the budget

Sign up for the October 25 webcast
Sign up for the October 31 webcast
Sign up to learn about future Perseus Webcasts

eDigitalResearch today released a press release highlighting their relationship with U.K. based department store John Lewis, who is using their system to collect and report on real-time customer satisfaction related issues.

eDigitalResearch’s Satisfaction Suite collects customer purchase information from the John Lewis database and emails surveys out to customers, tying in all the purchase information with customer responses. Monthly reports are produced which can be cross-tabbed and filtered in terms of product groups or particular divisions.

John Lewis’s particular satisfaction solution uses elements of both eDigitalResearch’s eMystery Shopper program and their eCustomerOpinion program. eMysteryShopper specializes in the in-depth and structured studfy of website usablity, functionality, and customer service using a eDigitalResearch’s panel. eCustomerOpinions also provides feedback, but from directly from randomly selected web site visitors.

View the press release at PRWeb.
Learn more about John Lewis.
Learn more about eMystery Shopper.
Learn more about eCustomer Opinions.

MarketTools has introduced Zoomerang Online Focus, a full-service online focus group service that provides participant recruitment, moderation and reporting. The service looks fairly easy to use and similar in process to traditional focus groups: First Zoomerang creates a discussion guide for you to review and approve based on the objectives and parameters you provide. Next, Zoomerang will recruit respondents either from your in-house database or from the "Zoomerang Sample" of 2.5 million consumers — note that respondents will be selected to match the needs of your study.

Next, a live Zoomerang moderator conducts your focus group online using their own proprietary focus group/chat software. You can watch the focus group as it proceeds from your own computer. Note that the Zoomerang Online Focus interface is fairly interactive and allows participants to write, type and draw on a white board that everyone can see and further allows them to participate in exercises devised by the moderator.

Shortly after the focus groups are over, Zoomerang sends you a final report with recommendations and conclusions, as well as a transcript of all chat discussions, recording of online comments and full audio feedback.

Pricing for Zoomerang Focus Groups is as follows: one group costs $5900 plus incentive ($240 – $800). If you want to use members from Zoomerang’s panel, the cost is around $1200. For two sets of focus groups, the price drops to $11k + incentive ($240-$800) + optional Zoomerang panel ($1.6k).

MarketTools is certainly not the first company to offer online focus groups — other companies include e-Focusgroups, iTracks, GMI, and others.

Katie Fehrenbacher of GigaOM posted a note today about Embrace, a UK-based mobile survey startup that has recently introduced MSurvey, a service for running questionnaires via cell phones. Embrace’s mobile solution is particularly useful for conducting very short surveys such as opinion polls, customer satisfaction surveys, employee satisfaction surveys, and ad hoc market research.

To maximize response rates, Embrace can collect data through three different method:

• SMS surveys: universally accessible from all types of mobile phones, allow for reverse billing (so your users don’t have to pay to take your surveys) but can only accomodate a limited number of question types;
• Phone Web Browser: Most new cell phones have this capability; surveys can be branded, and a range of question types are available.
• Downloadable application: Survey can be downloaded to the user’s phone and the survey can be completed when the user is out of signal area (like downloading a game to your cell phone).

Data can then downloadable in the Survey Interchange Standard format or XML.

Read Katie’s article at GigaOM.
Read more at the Embrace web site.

Stefania Viscusi of TMCnet recently posted an article that offers ten top considerations for selecting call center software for your company. Although not completely on topic, it does have some research ramifications (one would hope that your call center is in some way integrated into the collection of customer satisfaction data, which is definitely marketing research related).

Read the full article at TMCnet.
Props to Jim Berkowitz for finding this article first!

Customer research, taken out of its appropriate marketplace context, can be extremely misleading. Consider the scenario presented by Lior Arussy of the Strativity Group who in a recent DestinationCRM article where a research firm, after conducting a study to help a client indentify key loyalty factors to build greater customer relationships, came back with a finding that the most important thing the company could to to retain customers was to "excel in invoicing."

Lior argued (and I agree with him) that from the customer’s perspective accurate, on-time invoices — like clean bathrooms or safe rides at a theme park — aren’t reasons that most customers are going to do business with you. Sure, they’re important to maintain and ultimately speak to the gestault of how people perceive your business (nobody wants inaccurate invoices) but nobody is really going to choose you over your competitor if your greatest claim to fame is that you have the most accurate billing system in the business.

Says Lior:

"The goal of customer experience is not simply to stop upsetting people, it is to delight them and maximize revenues and loyalty. It is essential that market research surveys–and the client companies they purport to help–target and measure true experiences that help competitive differentiation."

To derive insight from research takes more than just good methodology and execution — it also requires an understanding of the business that your in and enough knowledge about your customers to be able to interpret the results in such a way leads to meaningful, actionable findings. In other words, you can’t simply leave it up to your research firm to go out, do a survey, and report back with results that you can immediately integrate into your business. You also need to bring to the tables your own experience and your own knowledge of the business at every stage of the research in order to ensure that the results that you get make sense in the context of your work.

Read Lior’s article at DestinationCRM.

Palo Alto, California based Attensity recently announced the release of version 4 of its text analytics suite which includes new methods of searching, querying, charting and graphing freeform text dynamically in an "easy to use" browser based interface. This release is designed to improve usability, add functionality, enhance results, streamline collaboration and increase interoperability with other applications.

Attensity 4 combines Attensity Server (used for text extraction); Attensity Discover (freeform text exploration); Attensity Analytics (quantitative analysis of text); Attensity Text Search and Attensity Manage.

Attensity 4’s new Text Search module allows users to test hypothesis by running ad hoc searches to determine if documents contain keywords, concepts, issues, or problems they are seeking to understand. It also allows users to easily find documents of interest employing advanced search features like fuzzy search, phrasal search and relevanc scoring; and it allows users to delve deeper into documents of interest by running advanced text analysis against documents returned in search results.

Read Stephen Swoyer’s aritcle in Enterprise Systems.

Read the full press release at the Attensity web site.

For those of you tired of simple multiple choice radio box, check box kind of online survey questions, RelevantView has created a new online survey tool that offers the ability to have users participate in an online card sort much like what they would experience if they were participating in a real-life exercise.

Although I can’t tell from the image precisely what technology is being used to power the sort, it does have a very Web 2.0 feel to it that hopefully gives a sense of the types of fancy new user-friendly online survey technologies we can expect to see in the future.

Read the press release.