Jared Heyman, president of Infosurv, recently published a white paper to Knowledgestorm.com entitled "The Benefits of Hiring a Full-Service Online Research Firm — Software vs. Online Research Firms," in which he pretty much "proves" that survey software should only be used for very simple, unimportant polls that don’t require a thoughtful methodology or unbiased execution. He says:

Online survey software is well-suited for "quick and dirty" type surveys where  directional quantitative or qualitative is needed. These surveys often run very short, with 20 or less items, and do not require the statistical accuracy that a full-service research firm can assure. An example may include a survey of twenty PTA parents to see what they would like the school to accomplish that year for their children, where only a rough tally of program preferences is required. Such surveys also do not require complex response validation, piping, or branching logic.

Although I agree with Jared on some points, I do think some of his assumptions don’t necessarily reflect the reality of today’s online survey software (many packages make it quite easy to validate responses or branch from question to question) and doesn’t take into consideration the access that many companies now have to thousands if not hundreds of thousands of customer e-mail addresses, along with the demographic data necessary to pull a meaningful, relevant sample. Sure — if you have no idea what to ask or if you don’t feel comfortable formulating questions — or even if you don’t have access to a good respondent pool — then it definitely makes sense to work with a professional agency.

I wonder if there isn’t some middle ground for companies and organizations who have the in-house know-how to write acceptable questions but need some help analyzing the data; or maybe the analytical capabilities are in-house but a broad-based response panel is still needed because the in-house list isn’t adequate. In other words, given the current economic environment where companies can’t afford to do a lot of research if there isn’t a sizable market out there for research firms that are willing so sell their surveys a-la-carte at reasonable prices…as far as I know there are a few companies who will do this, and then there is always www.guru.com.

Jared’s article is an interesting (albeit a little biased) read and covers many of the issues that you will have to think about when you conduct research yourself instead of hiring a firm.

Read the white paper at KnowledgeStorm (registration required).