Asking the right people to participate in your research study is just as important as designing and asking the right questions. You simply can’t have a successful market research project with a rushed recruiting effort at the start.
Sometimes, this means using a research vendor to assist with fielding the right participants. Vendors such as panel providers and expert networks are frequently tapped to identify respondents for research studies. But if you are using a research vendor and still throwing out a large number of respondents or are having trouble making sense of the data, it’s time to dig into the problem. And that may mean it’s time to switch to a new sample provider.
At Cascade Insights, we have quality checks in place to ensure our sample providers are fielding the right respondents. One element of these quality checks includes verifying that the respondent’s job title, past work experience, and knowledge will be a good fit for the research effort.
However, in recent years we have noticed an increase in the number of bad actors taking our surveys. At one point, we were disqualifying over 50% of respondents, oftentimes even up to 80-90%. This overall decrease in respondent quality led us to wonder about the percentage of those who did make it through our screen tests. Were they actually good respondents, or were they just missing the traps we had in place?
To answer this critical question, we investigated the panel providers we were using. Ultimately, we decided to discontinue many of our panel provider relationships and switch to expert networks instead. Below, we’ve laid out the issues that we encountered with panel providers to date and our experience with switching to expert networks as our primary source of research participants.
The Plummeting Panel Provider
Panel providers collect and manage groups of people who are interested in taking surveys and participating in research studies. Panel providers then mine this pre-existing database of participants to field a survey. Unfortunately, when this existing database runs dry, panel providers tend to mine the databases of other 3rd party panel providers. This shifting of responsibility presents a quality risk, as the source of the sample now comes from untrusted 3rd parties, many of whom you may be unfamiliar with.
Obviously, one clear way to address this problem is to write an effective screener. An effective screener can “catch” poorly chosen respondents and prevent them from skewing a study’s results.
Screener Example for B2B Tech
Which of the following cloud vendors have you heard of? (Check all that apply)
[ ] Amazon Web Services (AWS)
[x] Digital Cloud Corp (DCC)
[ ] Microsoft Azure
[x] Facebook Launchzone
[ ] Google Cloud Platform (GCP)
[x] Apple iBiz
If the participant chose one of the fake options (Digital Cloud Corp, Facebook Launchzone, or Apple iBiz), they were cut. Or,
Which of the following cloud vendors have you heard of? (Check all that apply)
[x] Amazon Web Services (AWS)
[ ] Digital Cloud Corp (DCC)
[ ] Microsoft Azure
[ ] Facebook Launchzone
[ ] Google Cloud Platform (GCP)
[ ] Apple iBiz
Which of the following cloud vendors are you using? (Check all that apply)
[ ] Amazon Web Services (AWS)
[ ] Digital Cloud Corp (DCC)
[x] Microsoft Azure
[ ] Facebook Launchzone
[ ] Google Cloud Platform (GCP)
[ ] Apple iBiz
If the participant indicated they were using a platform they weren’t aware of, they were disqualified.
Initially, this simple system of well-constructed screeners successfully helped us to weed out most of the bad actors. Once the “Great Resignation” hit, however, the quality of respondents we were receiving dropped substantially.
An Out of Date Database
According to CNBC, 4.4 million Americans quit their jobs in February 2022, and more than half of them switched their occupation or field of work. People were in new positions with new responsibilities and maybe not even in the same sector as before. If they were a key decision-maker before then, are they still making the buying decisions now? Or if they were working for an enterprise company yesterday, were they leading a startup today?
This amount of job churn broke the panel provider model as their internal processes for keeping respondent databases up to date couldn’t move fast enough. And because panel providers rely on their database to field surveys, the massive job mix-up meant the data was no longer valid.
Because of the ongoing problems we saw with panel providers sending us low-quality respondents, it was taking us much longer to complete surveys. At times we would fill a survey, stop and disqualify nearly all of the respondents, and then fill the survey again.
The final straw was when we added a question to the end of our survey asking respondents, “Who sent you this survey?” We expected they would say the panel provider we had contracted to field the survey. But, that wasn’t the case.
Respondents indicated that they had received our survey through consumer panel websites such as Qmee, Swagbucks, and Freecash.
This was a huge red flag. We knew the audience we were looking for wasn’t spending time on websites that earned them 50 cents a survey. This ultimately led to us discontinuing our relationship with a number of panel providers, as the problems we identified were not isolated to a single panel provider.
Where can you use panel providers today? For simple questions. Or those that tend toward a B2C focus. For example, How has COVID impacted small business owners? What do employees think of the wellness programs provided by their employer? Which color pallet do you prefer for Valentine’s Day M&Ms? But those aren’t the kinds of surveys we do and probably aren’t the kinds of surveys you need.
So, what is the solution? What choice do you have besides screening, vetting, and try, try again when it comes to panel providers? Consider switching your source of respondents to expert networks like we did.
For Quality Respondents, Switch Into Expert Mode
Expert networks connect clients with experts who are paid for sharing their industry or job role related expertise. They facilitate the matching of expert respondents to a researcher’s interest area. They schedule the interviews, create processes to ensure that any information exchange is compliant, and handle the payments.
Since we have switched to a focus on expert networks, we have noticed an increase in our data quality. Expert networks aren’t just leveraging an internal database or those offered by 3rd parties to get the quickest response at the lowest cost. Instead, they are hands-on in ensuring the participants who take our surveys are who we are looking for. This includes the right job title, industry background, work experience, and skills.
We still have our quality checks in place while using expert networks, but we have seen a significant decrease in the number of respondents we disqualify. The respondents we do get are, in the large part, legitimate. We don’t see many incorrect answers tied to platforms that don’t even exist or conflicting answers saying they are using a platform they aren’t even aware of. Additionally, we have the ability to get in contact with our respondents to confirm their identity or follow up on a question. An option that wasn’t available to us when we leveraged panel providers in the past.
Yet, this improved data quality does come at a cost. Expert networks are about 4-5 times more expensive than panel firms on a per respondent basis. However, we are certain that the right people are answering the right questions, which is critical. After all, does it really make sense to spend tens or hundreds of thousands of dollars on a survey of the wrong people who give the wrong answers? We don’t think so.
When it comes down to it, you need to be sure that your research vendor is providing quality that is worth it. If you are questioning why you are disqualifying nearly all of your survey respondents or why you have to re-fill your survey for the third time, it’s time to start asking your research providers and their associated panel providers some hard-hitting questions.
Key Questions Research Firms Should Be Able To Answer About Their Panel Providers
- Do your panel provider(s) provide all the sample for our study, or do your panel providers use 3rd party sources for their respondents?
Concerning answer: In addition to our own database, our panel providers use trusted and vetted partners to meet the quota requirements across segments.
If a panel provider indicates they are pulling in partners to field your survey, expect some rough patches in your data. Generally, when a panel provider says they are pulling in partners, they are leveraging consumer panels (Qmee, Swagbucks, Freecash, etc.) to save on project costs.
Better answer: Our panel providers may have some people in their database from past projects, but if needed, they will hand-recruit 100% of the respondents. They never get participants from 3rd parties.
- How do you ensure the research participants are who they say they are?
Concerning answer: We implement multiple layers of quality control to ensure that low-quality respondents are identified and removed from the survey.
Quality control isn’t inherently bad, but if it’s a cover for bad business practices, such as shuffling your project to the lowest bidder, it’s a red flag.
Better answer: Our partner recruits participants based on their LinkedIn profile or other public sources. They then require those people to confirm their identity through their company email address. A process that is standard for many expert networks.
- Can we follow up with the respondents that have participated in the survey?
Concerning answer: It’s not something we typically do, but we can ask our panel provider.
In reality, panel providers don’t let you speak directly to participants. You have no way of actually identifying the respondents are who they say they are.
Better answer: Absolutely.
Expert networks hand-pick respondents for every survey, so you have the ability to reach out and confirm their identity or ask them a follow-up question.
- What percentage of the cost per complete go to the participant as an incentive?
Concerning answer: It depends on a variety of factors and varies from project to project. There isn’t a good way to give a specific amount.
If a panel provider can’t tell you how much they will incentivize or even give a rough estimate, you should be concerned. That means they are probably fielding your survey to consumer panel websites and don’t want to admit that they are paying participants 50 cents a survey.
Better answer: About 2/3rds.
Expert networks may charge $200 per completed survey, but 50-70% is going towards their overhead costs, and 2/3rds is going towards an incentive for the participant.
Don’t Settle for Feeling Queasy
Why spend time and resources on recruiting quality respondents? Because getting the wrong answers from the wrong people simply makes no sense.
So, here’s a homework assignment: Consider the recent surveys you’ve had a market research firm conduct on your behalf.
- Did your market research partner(s) struggle to get quality respondents?
- Did this struggle lead to project delays?
- Were you concerned with how respondents answered the questions you posed?
- And did the market research vendor put the right amount of effort into managing sample provider relationship(s) over time?
If any of these questions make you uneasy or even queasy, it’s time to validate that you have a survey, results, and recommendations you can trust. And if you have a trusted market research partner who can design, test, validate, and field that survey for you.
With 15 years of experience in B2B tech market research, Cascade Insights can help ensure your research is of the highest quality. For more information on B2B market research, visit What is B2B Market Research.
Special thanks to Sean Campbell, CEO, Scott Swigart, President & CTO, and Tyler Honsinger, Director of Research, for advising on this piece.