SPSS recently put out a news release (which has been picked up by at least two news sites CRM Today and TMCnet) whose sole purpose appears to be to scare companies away from using the plethora of survey research tools in favor of their multi-modal survey system. How can you begin to know what your customers are thinking, reasons the release, if you only ask those who are online when hardly anyone is even using the Internet these days?

"Web-based surveys may appear to be less intrusive and easier to conduct, but without pen and paper or a good ‘old-fashioned’ telephone, organizations miss the opinions of many, including those without a computer, the forever and selectively computer illiterate and a large part of the senior population that simply missed the tech revolution."

The release goes on to remind everyone that paper and phone survey are "in many cases essential, if organizations expect to present clients with the most accurate and complete view of customer attitutes and opinions." 

It then proves its point by referring to a recent Pew report:

"In fact, the Pew Internet & American Life Project recently found 49 percent of Americans only occasionally use modern gadgetry and many others bristle at electronic connectivity — the Internet."

Wow. Who would have thought that in 2007 more than half of the US population either doesn’t use and/or extremely dislikes the Internet. We all may need to rethink our online programs and go back to the phone banks, door-to-door solicitors and shopping malls many of us have mostly abandoned.

…but before we do, here are some stats not included in the SPSS release:

• Total US population is about 300 million people with 225 million of them over the age of 17. (US Census)
• There are 178.8 million web users in the US (comScore, June 2007)
• 71% of all adults are online (Pew)
• 87% of 18-24 year olds, 83% of 30-49 year olds, 65% of those 50-64 and 32% of those over 65 are online. (Pew)
• 73% of white, 62% of black, and 78% of English-speaking Hispanic are online. (Pew)
• 73% of people living in Urban/Surburban Environments and 60% living in rural areas are online. (Pew)
• 93% of those earning $75K+, 82% of those earning $50K-$74K, 69% of those earning $30K-$49K and 55% of those earning less than $30K are online. (Pew)
• Total number of households is 105.4 million (US Census)
• Almost 70% of US households have Internet access at home.  (Leichtman Research Group Q1 2007)
• 53% of US households have high-speed access (Leichtman Research Group Q1 2007)

The Pew study that SPSS refers to in their release is called "A Typology of Information and Communication Technology User." The study measures not whether or not people have internet access (as implied by SPSS) but instead tries to categorize people by the degree to which information and communication technologies are utilized and enjoyed.

According to the report, only 15% of the population can be characterized as "Off the Network" — that is, individuals with neither cell phones nor internet connectivity. They tend to be in their mid-60s, nearly three-fifths are women. Only 7% have college degrees (vs. the US average of 27%) and only 4% earn over $75K a year (vs. the US average of 22%). They are the heaviest users of "old media" such as radio and TV but do not have the inclination to try new information and communication technology.

Obviously, not everyone is online and if you’re looking for a particularly special group you may want to revert to paper, pencils and phones. However, I’m thinking that for most purposes you’re going to be able to find who you’re looking for online.

But the implication of the SPSS release is that unless you use (expensive) multi-channel research techniques (provided by them?) you will be collecting bad information and misleading your clients. This isn’t true and is in fact extremely misleading.

• Read the SPSS news release
• Read Pew Report "Typology of Information and Communication Technology Users"

Martin Day, a Director of SurveyGalaxy, wrote an interesting article entitled "20 Top Tips to Writing Effective Surveys." The article is simply laid out, easy to read, and offers some pretty simple straightforward tips for writing online surveys (or offline surveys for that matter). My favorite tip (of the 20) is "Ensure that the questionnaire flows: whenasking questions group the questions into clear categories as this makes the task of completing the survey easier for the participants."

Sometimes it seems that many of the folks writing surveys — even the professionals — don’t seem to get it that the people taking the survey are for the most part volunteers (unless you’re paying everyone who takes your survey and not doing some kind of a drawing, almost all of your respondents are effectively volunteers) and if you don’t make the experience interesting and perhaps even fun then it is unlikely that anyone is going to go to the trouble of finishing the survey.

Not every survey can be fun. Some surveys are on boring topics. Some surveys use complex methodologies that make it difficult to create any kind of positive user experience. But it seems to me that it is important for us to at least try. If we’re going to ask our volunteers to give us their time and their opinions, it seems that the least we can do is try to make the experience at least somewhat entertaining and interesting.

To that end, in addition to Martin’s article below, I also present you with a link to a page on SurveyGalaxy which offers a list of the most highly rated (i.e., most interesting) surveys available on SurveyGalaxy as rated by respondents. Note that these aren’t always the prettiest survey in the world or the most interactive — but something about them has made respondents give them high ratings.

• Article: 20 Top Tips to Writing Effective Surveys
• SurveyGalaxy List of Best Public Surveys

Is it enough to survey 100 people or are you only going to get useful results if you survey 1,000 people? The answer, unfortunately, really depends on the questions you are asking, the likely results, and your preferred "margin of error" (the +/- 3% or +/- 4% you see posted with most survey results). You basically need this information so you can reliably know whether that 4% difference between the two bars on your graph mean anything or not.

Personally, when I’m conducting an online survey I tend to prefer a sample of 1,000. In an overall sense it is usually overkill, but it usually allows me to segment the results in a number of different ways — I can break the results down by age group, gender, income, etc — something I couldn’t necessarily do if I started from a much smaller sample. I suppose it is the luxury of having access to a large respondent base — I can afford to oversample. Believe me, if I were paying $10 a response (like what I sometimes have to do when I rent a panel of people in another country) I am much more conservative in my sample sizes and pay really close attention to my margin of error and the needs of the study.

There are some web sites out there that make margin of error more understandable. The Red River College Marketing Research blog recently pointed to an article at www.isixsigma.com entitled "Margins of Error Made Easy!" which I found to be worth reading.

There are also several sample size calculators out there (you can find them by typing "sample size calculator" into Google). One that appears especially handy is at dssResearch.com. Grapentime.com offers not just a sample size calculator, but also a sample size calculator for attribute ratings (in other words, it tells you the minimum sample sizes you need for different type of metric mesasurement scales).

Margins of Error Made Easy at Isixsigma.com
Sample size calculator at dssResearch.com
Sample Size Calculator for Attribute Ratings at Grapentine.com

Is it bad to offer incentives to people who complete your surveys? On more than one occasion I’ve had to answer this question for clients and others who were concerned that by offering a potential respondent some kind of perk for participating, you might somehow corrupt the data: either as a result of skewing the response base or by somehow influencing the respondent (perhaps to give you a better score, because they like you so much for giving them a free cup of coffee or an entry in a drawing to win a free Ipod).

My personal opinion is that there is nothing wrong with incentives — in fact, that they can be quite beneficial — as long as you are thoughtful about what you are offering and to who. For example, if I want to find out how interested people are in visiting a particular theme park, I probably should offer as my prize two tickets to the park in question — the only people who will take my survey are those who want the theme park tickets, and those will only be people who like theme parks — and so my results will, in fact, be pretty skewed. On the other hand, if I offer something more neutral — such as cash — then I should get a pretty broad based response since, as far as I can tell, liking cash has no bearing on whether or not you like theme parks.

If you need a more credible source than my personal opinion, a variety of studies have been conducted on the subject of incentives and the impact they have on research. I recently came across a literature review by Eleanor Simmons and Amanda Wilmont ("Incentive payments on social surveys: a literature review") in which they looked at incentives from many different angles, including the impact on response rates; monetary v. non-monetary incentives; value of incentives; differential incentives (which is when you only offer to pay incentives to people after they refuse to take your survey); the effect of incentives on interviewers; incentives and data quality; effects of incentives on sample composition; and the effects of incentives on long-term expectation effect.

Read "Incentive Payments and Social Surveys" here.

I was poking around in Google today and came upon an interesting article in the October, 2000 issue of hte CustomerSat.com monthly newsletter (appropriately entitled "CustomerSat.com Connections" about the relevance of using top box ratings, bottom box ratings and mean scores.

The article argues that it is impossible to get a true understanding of your customer’s satisfaction by simply using one of the three metrics, and that the best way to really understand what is going on is to use some combination of at least two, one of which will be the mean score.

In terms of determining whether it makes more sense to look at the top box or the bottom box, it suggests examining how changes in either box correlate with the overall outcome.

What is most fascinating and useful about the article is actually its description of "Non-Linear Effect Analysis" in which it describes how customer loyalty tends to rise not linearly, but exponentially. In other words, customers who give a "9" out of 10 instead of an "8" out of 10 are exponentially more loyal.

Read the full article at CustomerSat.com
See all of the online resources offered by CustomerSat.com

Research provider Knowledge Networks (Menlo Park, CA) has published an interesting nine page white paper entitled The Decisions Maker’s Guide to Online Research. Although the document is clearly geared towards illustrating how the Knowledge Networks panel methodology is better than others, the pamphlet does provide a thoughtful framework for deciding between web-based surveys with self-selected panels, web-based surveys with probability-based internet panels, mall intercepts, and mail surveys.

It identifies two types of online panels:

• Self selecting panels, which anyone can join — a "convenience sample" of the internet, and one that is likely to contain "professional respondents" and possibly even competitors trying to get insights into your secret plans;
• Probability-based Internet panels, which are (painstakingly) built by randomly selecting people, calling them, and then inviting them to join the survey panel.

One of the primary sources of data for the Knowledge Networks publication was a study conducted by the Stanford Institute for the Quantitative Study of Society entitled "Comparing the Results of Probability and Non-Probability Sample Surveys." The study begins by acknowledging that in general, the same survey conducted by two firms with the same methodology will usually yield comparative findings. However, most studies that led to this conclusion focused on surveys conducted in the same mode with comparable sampling methods. The folks at Stanford wanted to find out what would happen if the mode and sampling methods differed.

Nine data collection firms participated in the study: seven of whom use a self-selecting, volunteer sample (self selecting panel) and the other two who used probability-based panels (one that used a probability based telephone sample, the other (Knowledge Networks) that used an Internet-based probability panel). Each data collection firm asked their respondents the same set of questions, and the results were compared against benchmark probability-based responses.

The findings of the folks at Stanford were that the results were "remarkably comparable" across the board. Knowledge Networks had the most accurate findings, followinged by SRBI (telephone survey) and Harris Interactive (volunteer sample) who tied for second place (all of the others were about equally as accurate).

A few questions led to bigger differences between the self-selecting and the probability based methods: for example, volunteer respondents tended to be more comfortable using computers than probability-based respondents. Otherwise, however, it would appear based on the results of the study that a volunteer sample base will ultimately lead to results that are closely comparable to the more expensive probability-based sample.

Read the Knowledge Networks publication.
Read the press release announcing the publication.
Read the results of the 2005 Stanford study.

SPSS recently announced the release of SPSS Dimensions 4.0, the latest incarnation of its enterprise survey and analysis suite that does everything from helping you create surveys to analyzing the data to generating reports. Before looking into the new features introduced in version 4.0, I thought it might first be interesting to explore the basic features of the system. In other words, what is SPSS Dimensions?

SPSS Dimensions isn’t so much an individual software package as much as it is a platform of several independent software packages that are able to work together in a relatively seamless fashion. Sort of like how each of the programs within Microsoft Office (Word, Excel, Powerpoint, Access, etc) can work independently (and be purchased independently) but also work very well together. Like Office, most of the packages that work with Dimensions are published by SPSS — although the platform has been designed to accommodate the integration of software written by 3rd party developers (and several such packages do exist).

Some of the programs that work with Dimensions include:

• SPSS mrDialer – Automated dialing for phone surveys
• SPSS mrInterview – Create and execute online surveys
• SPSS mrInterview CATI – Create and execute phone surveys
• SPSS mrPaper – Create and execute paper surveys
• SPSS mrScan – Scan paper surveys
• SPSS mrStudio – Manage and manipulate data
• SPSS Desktop Reporter – Create tables from local data
• SPSS mrTables – Interact with tables on your desktop
• SPSS mrTranslate – Manage translations of surveys and reports
• Techneos Entryware – Collect data using handheld devices
• SPSS Base – Analyze data
• Clementine – Data mining
• SPSS Text Analysis for Surveys – text analysis & categorization

At the core of SPSS Dimensions is the Dimensions Data Model, a set of components (openly documented and supported) which allow for accessing information about questionnaires and respondent data. It also deals with keeping track of changes to the questionnaire (versioning), translating both questions and data from one format to another, and managing data stored in multiple formats and platforms.

Visual representation of the Dimensions Data Model
(from the SPSS presentation "Using the SPSS MR Data Model")

The table above describes the role of the data model well. Data can be collected from multiple sources. Instead of each collection program storing the data in its own database, it instead sends it to the Dimensions Data Model which puts it into its own special format. When another program, such as a data processing program or a data analysis program needs the data it requests it from the Dimensions Data Model using standardized request formats (that just about any program can use).

Consider a project in which you need to collect data using three different survey techniques including a phone survey, a web-based survey and a paper survey. Even though you’re going to ask the same basic questions in each survey, you are still going to have to develop three completely different questionnaires in order to compliment each of the mediums, which further means you’re going to have to program the survey three different times (perhaps four, if you consider that you may be using scanning software to read some of your paper surveys). 

Even though the question is the same, it needs to appear different
across modes and across functions (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

After you’ve finished collecting the data (using three separate data collection tools, all of which store the data in their own, separate proprietary format that exports into the frustratingly simple CSV format, you’ll then have to combine all of the data into one file which you’ll then need to clean and prepare for analysis. Following analysis, you’ll export your results into yet another program.

Using the SPSS Dimensions Suite (or more specifically, software that is integrated into the Dimensions Suite) makes the process go much faster by optimizing the mechanics of designing and fielding your questionnaire and analyzing and reporting on the data.

Dimensions reduces the time it takes to conduct a complex research project
(Source: SPSS presentation "Discover it with Dimensions")

SPSS Dimensions has been developed based on the notion of "Design Once, use Many" so once you have created your initial questionnaire (either using a simple, graphic user interface found in mrInterview or the more advanced script driven interface provided by mrStudio — either package will allow you to import the text of your survey from MS Word), you can then quickly (and easily) set it up to deploy using multiple modes (paper, web, CATI, etc).

Perhaps one of the most exciting features of SPSS Dimensions is its multi mode deployment capabilities. Most surveys today require some amount of programming to deal with skipping, piping, the incorporation of outside data, and other advanced options. Ordinarily, each mode would require its own programming. SPSS Dimensions is designed so that you only need to write the script once and it will work the same in each context.

SPSS Dimensions allows you to program your survey once
and have it work on multiple platforms. (from the SPSS presentation
"Improving Government Programs with Comprehensive Data Collection")

Dimensions not only helps you design and execute your survey, it also manages security, translating the survey into multiple languages, and manages multiple versions of your survey as well.

External databases containing participant details can be added at any time, and it can be used both in the survey and during analysis. Data from outside sources can be reviewed during the survey to check for inaccuracies, and it can even be updated based on responses given in the survey.

All of the data that is collected, regardless of how it is collected, goes into one SPSS Dimensions database where it can then be analyzed and reported on. Although Dimensions is an open platform that will allow analysis to be conducted in any program (it will export data into a variety of formats for other programs to use), the suite is optimized to work with several SPSS-published programs, such as mrStudio, mrTables, SPSS for Windows and Clementine. Results can then be automatically turned into interactive web-based reports or analyzed using Excel, Word or PowerPoint. Dimensions integrates all of the major capabilities provided by SPSS’s various data analysis packages, including SPSS Base for statistical analysis; Clementine for data mining, and SPSS Text Analysis for Surveys for text analysis and categorization as well as a variety of SPSS and 3rd party data collection and reporting tools.

Reasons to Consider SPSS Dimensions

• Powerful interviewing engine
• Open architecture
• Web-based user interface
• Easy to create surveys
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Write the survey once, use in multiple modes (phone, web, etc)
• Write the scripting/programming once
• Easy to program (similar to VB Script)
• Common data storage format/interface
• Translation capabilities
• Faster development and analysis time
• Works with (some) third party applications
• Scriptable (write your own scripts to work with data)
• Integrates well with SPSS
• Integrates well with Excel

Reasons not to use SPSS Dimensions

• Expensive
• Limited to Dimensions compatible tools
• Complicated to set up and integrate with existing systems
• Requires lots of IT support

Learn more at the SPSS web site.

About a year ago Tim Macer gave a presentation at the SPSS Directions Conference entitled “Weaving, not drowning: An update on take-up and best practices in Mixed- and Multi- mode research.” Long, perhaps even academic sounding title, but actually extremely relevant to folks trying to figure out how to conduct and combine multiple modes of research (phone, web, paper, etc). In his presentation, he agenda covered the following questions:

• Who is doing it, how common is it?
• Why are they doing it?
• Why are some other people not doing it?
• Which modes work best together?
• When does it make sense to switch modes?
• What impact does it have on the data?
• What are the technical requirements?

One really neat concept I hadn’t thought much about was the idea of having a respondent start the survey using one mode (perhaps paper or the phone) and then have them finish the survey in another mode (usually the web). This has been found to help out when it is hard to retain respondents in one mode using a particular data collection method (perhaps they don’t want to hang out in your store for 20 minutes, or maybe they just want to get off the phone).

Multi-mode data collection will become especially useful as we adopt more mobile survey solutions — perhaps have the user start with a WAP based survey and finish up with a web based survey when they get home.

Read Tim’s full article at Meaning (PowerPoint).

In my experience, the use of incentives, whether it is the opportunity to win a prize in a drawing/sweepstakes or points towards some kind of special reward, can make a big difference in the response rate to an online survey. But what to offer? In what context? Perseus is offering a free webcast on Wednesday, October 25 or October 31 at 1pm entitled "Learn the Secrets of Affordable Incentive Strategies" to help answer these questions.

The hour long presentation, led by Larry Nichter (EVP of Restaurant.com) and Brian Koma (VP of Services for Websurveyor) will include:

• Incentive best practices
• When to use incentives
• What types of incentives are most effective for increasing response rates, maximizing completion and improving data quality
• How to put incentive strategies in place that don’t blow the budget

Sign up for the October 25 webcast
Sign up for the October 31 webcast
Sign up to learn about future Perseus Webcasts

MarketTools has introduced Zoomerang Online Focus, a full-service online focus group service that provides participant recruitment, moderation and reporting. The service looks fairly easy to use and similar in process to traditional focus groups: First Zoomerang creates a discussion guide for you to review and approve based on the objectives and parameters you provide. Next, Zoomerang will recruit respondents either from your in-house database or from the "Zoomerang Sample" of 2.5 million consumers — note that respondents will be selected to match the needs of your study.

Next, a live Zoomerang moderator conducts your focus group online using their own proprietary focus group/chat software. You can watch the focus group as it proceeds from your own computer. Note that the Zoomerang Online Focus interface is fairly interactive and allows participants to write, type and draw on a white board that everyone can see and further allows them to participate in exercises devised by the moderator.

Shortly after the focus groups are over, Zoomerang sends you a final report with recommendations and conclusions, as well as a transcript of all chat discussions, recording of online comments and full audio feedback.

Pricing for Zoomerang Focus Groups is as follows: one group costs $5900 plus incentive ($240 – $800). If you want to use members from Zoomerang’s panel, the cost is around $1200. For two sets of focus groups, the price drops to $11k + incentive ($240-$800) + optional Zoomerang panel ($1.6k).

MarketTools is certainly not the first company to offer online focus groups — other companies include e-Focusgroups, iTracks, GMI, and others.

Customer research, taken out of its appropriate marketplace context, can be extremely misleading. Consider the scenario presented by Lior Arussy of the Strativity Group who in a recent DestinationCRM article where a research firm, after conducting a study to help a client indentify key loyalty factors to build greater customer relationships, came back with a finding that the most important thing the company could to to retain customers was to "excel in invoicing."

Lior argued (and I agree with him) that from the customer’s perspective accurate, on-time invoices — like clean bathrooms or safe rides at a theme park — aren’t reasons that most customers are going to do business with you. Sure, they’re important to maintain and ultimately speak to the gestault of how people perceive your business (nobody wants inaccurate invoices) but nobody is really going to choose you over your competitor if your greatest claim to fame is that you have the most accurate billing system in the business.

Says Lior:

"The goal of customer experience is not simply to stop upsetting people, it is to delight them and maximize revenues and loyalty. It is essential that market research surveys–and the client companies they purport to help–target and measure true experiences that help competitive differentiation."

To derive insight from research takes more than just good methodology and execution — it also requires an understanding of the business that your in and enough knowledge about your customers to be able to interpret the results in such a way leads to meaningful, actionable findings. In other words, you can’t simply leave it up to your research firm to go out, do a survey, and report back with results that you can immediately integrate into your business. You also need to bring to the tables your own experience and your own knowledge of the business at every stage of the research in order to ensure that the results that you get make sense in the context of your work.

Read Lior’s article at DestinationCRM.

For those of you tired of simple multiple choice radio box, check box kind of online survey questions, RelevantView has created a new online survey tool that offers the ability to have users participate in an online card sort much like what they would experience if they were participating in a real-life exercise.

Although I can’t tell from the image precisely what technology is being used to power the sort, it does have a very Web 2.0 feel to it that hopefully gives a sense of the types of fancy new user-friendly online survey technologies we can expect to see in the future.

Read the press release.

David Tebbut of IWRBlog (Information World Review) recently posted some interesting observations about Confirmit’s dashboard application, in which an online survey system is used to track customer attitudes in real time and report the results in an automatically updated "dashboard" application. The idea is to be able to provide useful research results instantly — as soon as they are relevant — instead of having to wait hours, weeks or days for results.

In my own experience, the greatest challenge to this type of a dashboard — which in some ways speaks to the potential to integrate customer satisfaction directly into a balanced scorecard type system in a meaningful way — is the ability to collect enough data on a regular basis to cause the "dials" on the dashboard to reflect something meaningful. 10 or 15 responses a day are simply not enough for a system that is meant  to be continuously updated.

On the other hand, there are applications where such a system might be somewhat useful and relatively easy to "keep fed." For example, if on the way out of the store there was a single question that customers could answer — either as they walked out the door or as they checked out — and if there were enough registers in the store — it might be possible to collect enough data to make the dashboard meaningful. Or maybe if there were a way to ask the question on customer cell phones as they walk out of the store (perhaps a little less realistic).

Read more about this article at IWRBlog.

Last week 30 of the top executives in market research met for a rountable at the Research Industry Summit for Improving Respondent Cooperation. It would appear that response rates of less than 10 percent are becoming more common, with reports from NOP Research indicating that just 0.25% of the population is supplying 32% of the responses to online surveys, and another report from Cambiar citing a recent ComScore Networks study indicating that 50% of all survey responses come from less than 5% of the population.

"It’s like the holein the ozone layer," said Shari Morwood, VP-worldwide market research at IBM. "Everyone knows it’s a growing problem. But they just ignore it and go on to the next project."

Although online research wasn’t particularly blamed for the problem, it was noted that while the internet channel makes it easier for respondents to complete surveys, there are now so many surveys out there that more and more consumers are tuning them out.

Read the full article at AdvertisingAge.

Mario Menti of msurveys.com recently posted a note on his blog describing how easy it was to create a survey, solicit responses, and then compensate the respondents in SecondLife. A very interesting, unique and possibily even useful experiment to be sure.

Second Life is a virtual community much like a massively-multiplayer online role-playing game (MMORPG) such as World of Warcraft, Everquest or Star Wars Galaxies except that instead of fighting monsters and completing quests your objective is…well…there isn’t really one. You buy property, meet people, simply fly around or create things. In fact, using the basic building blocks provided by the software you can create just about anything if you have enough patience and skill.

While most people build digital representations of physical things (like buildings, sculptures or stargates). Mario used his time and skill to create an online survey. In fact, he even set it up so that individuals who finish the survey are rewarded with in-game money.

Think about this in context with the recent findings of the Pew Internet & American Life project, where 52% of a broad-based, national sample agreed with the following:

"By the year 2020, virtual reality on the internet will come to allow more productivity from most people in technologically-savvy communities than working in the "real world." But the attractive nature of virtual-reality worlds will also lead to serious addiction problems for many, as we lose people to alternative realities."

The entire report (entitled "The Future of the Internet II") is fascinating and contains a number of interesting predictions for the future (along with what people thinking of them). Definitely worth a read.

I love Mario’s experiment. And while it is probably true that it is just a little too early to jump on this technological bandwagon (unless you’re trying to do a survey about virtual communities and the people who use them, it is probably going to be difficult to come up with a sample that is relevant to your research questions) I think the time won’t be so far off when data collected in environments such as Second Life is the norm.

Read about Mario’s Second Life survey experience on his blog.
Watch a video of someone completing the Second Life survey.