Will smartphones transform qualitative research methodology? Get an inside view from an expert who knows qual via smartphone inside and out.
In this episode of the B2B Market Research Podcast, Cascade Insights CEO Sean Campbell interviews Over The Shoulder Co-Founder Ross McLean. You can listen to the podcast or read an edited transcript below.
The following transcript has been modified for clarity and readability.
Sean
On this episode of the B2B market research podcast, I have with me Ross McLean, co-founder and president of Over The Shoulder.
Qual Via Smartphone
Sean
Ross, welcome to the podcast. Why don’t you give the listeners a summary of yourself and your company?
Ross
Sure. At Over The Shoulder, we do smartphone-based qualitative research.
Here is a quick background on the company. We were a bunch of qualitative researchers, innovation strategists, and ethnographers who did a lot of traditional qualitative research.
About nine years ago, we got really excited about smartphones.
In doing traditional qualitative, we had always suspected that speaking to people after a decision was made would not lead to the best insights. People don’t always do a great job of remembering the factors that led them to make a certain decision. They may not recall how they felt at a certain point in their pathway to purchase.
With smartphones, we saw an opportunity to reach people before their decisions get made.
We knew that smartphones would spend a lot of time in the pockets and purses of the people that we wanted to understand better. We said, “Gosh, that’s going to allow us to be there in the moments we’re missing out on right now.”
However, nine years ago, there just wasn’t any good software solutions to interact with participants in a project through their smartphones. Luckily, we realized we were just techy enough, we’d researched enough user experience, and we had enough software design experience to build such a solution ourselves.
We built a platform and did, I think, more than 400 qualitative research projects through smartphones over about three and a half years.
We’ve learned a lot and are very pleased with the results. We bumped into every sharp object out there and learned how to work around them from both an operational and software perspective. We gained a lot of nuance on how to ask questions and build refinements to do qualitative on the smartphone. Everything we learned helped us to build a seamless platform for qualitative research via smartphone.
About three years ago, as a group, the broad base of market research practitioners started to really take interest in smartphone based qualitative. We said, “OK, we’ve got a terrific software platform to enable this. We’ve got people who are great at designing projects in this space. We’ve got a help desk, community management, and operations stuff that we’ve been doing this for. Let’s stop doing direct, for-client projects. Instead, let’s make this a company that allows all qualitative practitioners and insight seekers to add smartphone qualitative to their tool kit. Let’s enable them to do that really well.”
Anyone who has strategic and qualitative skills can now work with us and execute smartphone-based projects really easily. Also, with us, they’ll have best-in-industry standards the first time out of the gate.
Participant Experience
Sean
What does the technology stack or infrastructure look like? What does a research participant see?
Ross
In terms of technology, it’s a native app for Android and iPhone.
Some of the most important things we consider for a positive participant experience are:
- Ease of use.
- Visual appeal.
- Affirmation of answers.
- Participant management.
You have to make it easy and even fun for people to pop out their smartphone and tell you what they’re experiencing right then and there.
If participants have a great user experience, you’re going to get great stuff.
We’ve spent nearly eight years trying to make the participant experience as smooth as possible.
This is how it works:
- The app is easy to download.
- Participants log in one time.
- Participants are given a briefing on how to use the app and what is expected of them during the project.
- Instructions are given for photo, audio and/or video capture. These instructions are designed to make it very easy for participants to use the app. We’ve got it to where if we want people capturing video, they do it within the app using their native video recorder. With the Over the Shoulder app, they literally just hit a button. They see the question that they’re being asked in the viewfinder of their camera or video recorder so that it’s there for them to reference and remember. They just hit a record button to answer.
- When they are finished with that assignment, they hit “I’m done.” All of the uploads are managed in the background for them. They don’t have to leave the app to go and take a photo using their smartphone’s camera and then come back to the home screen, then back to the app, etc. We took all of that out, to make it smooth and intuitive for participants.
Researcher Experience
The other area we put a lot of focus on is the analyst experience.
You get a ton of material when you’re asking people to pull out their smartphone and do an assignment which may involve audios, visuals, and photos, a choice every time they interact with a piece of software, or a response when they have a moment of frustration with a system or tool.
This generates a lot of data. We had to make it extremely efficient to consume, transcribe or translate, and tag it up so that the researcher can find stuff that matters quickly.
What we’ve learned is that for analysts to have a positive experience, the app needs to present information in a way that is quick and efficient. We also had to provide tools that allow researchers to approach their analysis the way that the qualitative mind approaches it.
Since we already were qual practitioners and spent a lot of time talking to other qual practitioners, we had a good understanding of the nuances of how qualitative analysis gets done.
We wanted to build the digital version of the way people were already doing qualitative analysis and give it super powers.
It has always been important to keep the app intuitive and simple for anyone who knows qualitative insight and strategy.
Sean
What does the experience look like on the backend for qual researchers?
Looking through that much video, audio, and multimedia must be time-consuming not only for the researcher but, I imagine, for the client as well. They’re obviously not going to sit and watch 10-20 hours of video.
Ross
Absolutely not. And sometimes even expecting them to spend an hour on it is generous. Instead, researchers often need the five-minute video capture that will convey the story and the insights that will get people interested to hear more. That’s the level of distillation you need to be able to get down to.
To do proper qualitative research, you’ve got to have someone who’s reviewing. A lot of the insights come from things that are non-verbal, so it was important to us to have someone looking at everything.
A participant may show you through their body language, “Hey, this piece of machinery is interrupting my workflow because I’m trying to operate it with my left hand and my right hand has to be doing something else.” That would indicate that the machine needs to be designed in a different way. You need to be able to see that, not just read a transcript of it.
It was hugely important for us to make the digestion of information through Over The Shoulder as efficient as possible.
The reviewer can theme, tag, and rate the qual data in real-time when watching or listening. That way, you can easily point to specific instances if asked.
This allows for several layers of analysis. Here are a few examples. You could only show videos that:
- Have been rated 3-4 stars for quality.
- Deal with themes related to usability.
- Have been tagged as of particular interest to a certain client.
Also, it’s easy to share this information quickly. You can download it to include in a report, or edit it into a video presentation, etc.
When you’re dealing with smartphones, everything is coming back in different formats and different codecs depending on each participant’s device. We have to re-encode it and, obviously, secure and encrypt it while it’s in transfer because a lot of this stuff is highly confidential. However, we also make it readily accessible, in standardized formats, and easy to edit.
The B2B Benefit
Sean
Tell me about how Over The Shoulder is used for B2B research. What are some of the more interesting examples you’ve seen of how the solution could be used in a B2B environment?
Ross
So roughly one in five of the practitioners we enable to do smartphone-based qual is using it in a B2B context.
Most researchers use our platform to better understand:
- How big decisions get made in the moment.
- The pain points of software in daily use.
The studies may have to be designed a little differently, but we have people looking into things like how to use mobile apps in a particular context, such as time or project management. These are all fantastic things to be able to do with smartphone-based qualitative, because those are things, that, by definition, happen in the moment.
Say you’re working on a mapping system that will be used by a sales organization or a bunch of technicians who will use certain mobile software to connect to that mapping system. To give two quick examples of the insights Over The Shoulder could bring to this scenario, our app could show:
- When and how people use maps on their smartphones.
- Frustrations users have with using different mapping solutions.
Over The Shoulder allows this feedback in the moment. Whereas with an interview, you’d have to ask people to remember the stuff they did earlier in the week or even further back. No one has perfect recall, so information derived from memory alone will always be slightly flawed.
With our platform, you can say, “Hit this button every time you are interacting with a piece of mapping software, tell us what you were trying to do, tell us what this piece of software did really well, and tell us the things you wish it could be doing.”
I’ll give another example of the benefit of smartphone-based qualitative research in a B2B scenario. Say you work in logistics in a shipping company. Your customer is interacting with some of your software or your competitor’s software, but they’re doing it seven times a day at totally random times. Over The Shoulder lets you tell them, “Every time you need to ship something, we want you to hit this button on your smartphone. Tell us the situation that you’re facing, the solution that you’re using, why you have picked this solution over the alternatives, how your chosen solution is working out for you, and how it could be improved.”
Privacy & The Platform
Sean
Let’s talk about the compliance and privacy concerns in a B2B context. That’s a problem that sometimes crops up even when you’re dealing with in-depth interviews. You have to promise anonymity to the respondents. Potential interview subjects will often say, “Well, I have some important concerns here about compliance and privacy, so I can’t have a conversation on that subject, etc.”
I imagine the minute you start asking to take video in certain environments like healthcare, or inside a data center, or something like that, you can end up with people just not willing to participate due to privacy concerns.
How often have you seen that become an issue? How do you mitigate those concerns?
Ross
Well, we help our practitioner clients do a ton of pharma work and that sort of sets the bar for personally identifiable standards and security.
We had to build a lot of encryption, monitoring and privacy protection into our system to be able to do that work. Obviously, that’s very helpful when you’re dealing with strategies and systems that are proprietary and confidential.
With data centers or manufacturing procedures, on the factory floor, you’re typically not allowed to be recording visual images. However, there are ways for people to interact that don’t capture video and photo. The person can describe the pain points of the process that they’re using with a recording of their voice, a text response, or just a multiple choice response.
Sometimes, we have to design very carefully to actively ensure not to capture stuff that is a breach of any kind of privacy protocol. There are ways you can design studies to make sure that doesn’t happen.
You can also have it so that all the information collected is reviewed for quality and any potential PII or security breaches by somebody who knows what is and isn’t out-of-bounds before it is submitted. That way, say if a doctor or nurse inadvertently takes a picture that captures a medical record with personally identifiable information on it, that image can be hard deleted out of the system before anyone else is going to see it.
You can take extreme security measures, but more than anything it’s just being transparent with the people who are participating. Tell them what you want them to capture and what you want them not to capture. Make sure they understand that they are not being asked to operate in restricted areas or share information that is outside of their privacy policies.
You can say, “We want you to acknowledge that we do not want you to share this type of information. Sign your name here.” They can add a signature to make sure this is understood right within the app as they are doing the tutorial.
Clarity is important, but you can also make it very hard for them to capture confidential information that you’d have to worry about in any way.
Does Big Data Replace The Need For Qual?
Sean
All fair points. It’s great to see you have a bunch of ways to gather information in sensitive environments.
You had a recent blog post about how big data misses the “why.” I completely agree with the post. Why don’t you summarize the message and your motivation for posting it on the Over The Shoulder website?
Ross
Several years ago, we wondered if big data and live capture behavioral data would be a threat to qualitative research- smartphone qual in particular. We thought that big data may have had the potential to answer some of the same questions. However, as we started to interact with companies that had a lot of behavioral information and big data at their disposal, what we found was really just the opposite.
Big data does an incredible job of capturing real behavior. It’s great for identifying whether there is something to pay attention to in terms of website interaction, social media topics of relevance, etc. Big data points you to the places you need to look into. It’s saying something’s going on here. There’s a quantifiable behavioral thing happening in this specific area.
Big data can point out what’s happening, but it doesn’t necessarily explain why it is happening.
Big data was much more complimentary to smartphone qualitative research than we had feared. In a way, big data points out qualitative objectives for you really well.
Smartphone qual then lets you understand with pictures, audio, and video what the human experience of this observable, behavioral phenomenon is. That’s what points people to understanding how to solve pain points.
Sean
I completely agree.
One of our larger clients had an employee who gave a talk at an industry conference. This particular client of ours owns a lot of very well known SaaS solutions. The individual giving the talk, a director of project management, I believe, was telling the audience, “I have all the telemetry in the world. I have way more telemetry than I had when the solution was on premise. I have all this data on who clicked what part of the interface, how long they spent on a given part of the interface, how much time they spent with the solution all up, etc..” His next slide was, “But we have no idea why they spent the time they did.”
Ross
Exactly. Problems come up when you don’t have the context around a decision.
For example, say we build into our website something that prompts you to answer a specific type of question. People click on it once and then they never come back. What did we get wrong in this scenario? Those are things that big data can’t necessarily answer. Big data does an amazing job of helping us realize, “Here’s something that’s not working the way we thought it would at all. We need to go and understand that.” Then smartphone qual can step in to explain the reasons behind the behaviors.
Sean
Thank you for being on the podcast and best of luck to you guys with the solution. It sounds really well crafted. If folks want to check it out, they can visit your website at overtheshoulder.com.
This podcast is brought to you by Cascade Insights. We specialize in market research and competitive intelligence for B2B technology companies. Our focus allows us to deliver detailed insights that generalist firms simply can’t match. Got a B2B tech sector question? We can help.
In many cases, B2B companies target much narrower markets than B2C companies do. A bottle of ketchup can be marketed and sold all over the world. Whereas B2B solutions, from our experience, may have a total addressable market of 100,000 companies or far less. This is a huge difference from a B2C product that could legitimately be sold to any US consumer who might walk into a grocery store.
Qualitative research can often answer B2B questions much more effectively than a survey. It’s a good compromise. While you won’t have the ability to project the findings from a qualitative effort to a broader population, you will be able to answer many questions and have confidence in the results.
