Know Thyself: B2B Usability Testing

Know Thyself: B2B Usability Testing

Sean Campbell
Authored bySean Campbell
Isa Gautschi
Authored byIsabel Gautschi

You can’t be unbiased when comparing your company’s offering to your rivals’.

We get it, you’re proud of the work you do and the innovations of your team. Of course, your offering is No. 1 in your eyes.

But… you do have to know where you rank. Building effective strategy depends on it. It’s just too important a matter to leave up to wishful thinking.

Luckily, B2B usability testing can give you an objective view of your pain points and competitiveness.

However, designing B2B usability studies requires finesse. You have to have the proper technology, access and, importantly, knowledgeable research participants.  In this episode of B2B Revealed, we brought in MeasuringU Principal Jeff Sauro to explain how usability testing differs in a B2B context.

Know Thyself: B2B Usability Testing

In this episode of B2B Revealed, we brought in MeasuringU Principal Jeff Sauro to explain how usability testing differs in a B2B context.

Listen to the episode…

[sc_embed_player_template1 fileurl=”″]

Or you can read the edited transcript that follows.

Cascade Insights CEO Sean Campbell

Today I have Jeff Sauro with me on the podcast. Jeff is a six-sigma trained statistical analyst and a pioneer in quantifying the user experience. He has authored and co-authored five books including Customer Analytics for Dummies, and Quantifying the User Experience: Practical Statistics for User Research. He’s also the founding principal of MeasuringU. Jeff, welcome to the show.

MeasuringU Principal Jeff Sauro

Thanks for having me, Sean.

So, Jeff, I reached out to you because you currently own the first spot in Google when someone searches “B2B usability testing.”  The reason for that is that your team put together a blog post on doing usability testing for B2B that is pretty popular.

Before we get into the particulars of B2B usability testing, let’s define some terms.

Walk me through some of the key elements of a usability research effort and the goals of a typical usability study.

Why Study Usability

Like an in-depth interview, a usability test has a one-on-one focus. However, with usability testing, the emphasis is more on observation than on a dialogue. You’re examining attitudes and actions with usability studies. The idea is to see what users do with the system when you’re not around to help.

With this approach, you can discover pain points. You may learn that users have trouble with forms, entry, or general navigation. (These are common issues across B2B enterprise applications.)

Collecting metrics is another important component.  As we all know, if you can’t measure it, you can’t manage it.

In sum, usability tests probe:

  • Usability issues.
  • The current usability of the system.
  • After fixing issues, whether the usability has improved.
  • How usability compares in competing systems.

The Evolution of Usability Tests

Your website has a really nice infographic on a timeline of usability history. How has usability testing evolved over the last 20 years or so?

I think usability really got its prominence in the mid-80s. In that time there was this idea that the usability lab was an extension of Stanley Milgram’s psychological experiments. You had this vision of white lab coats and PHDs conducting studies in antiseptic rooms with really expensive equipment. The perception was that only the IBMs of the world could afford to do usability studies. They didn’t seem as accessible to the emerging enterprise market.

It did start out that way. Usability tests used to require expensive equipment, highly-trained facilitators on location, the proper technology, and the physical space. Also, the participant had to be present.

As we moved toward screen sharing capabilities, things changed.

You could have participants share their screen as if they were in the room with to you. With that, many of the barriers for getting test participants were erased. We were able to take a step away from some of the complex systems previously necessary for conducting usability studies.

Another huge evolution is that usability tests no longer have to be moderated. We gained the ability to give participants a set of tasks as software records their screen. At the same time, we can track how long it takes for them to complete the tasks.

Today, you can collect data from hundreds of participants in just a matter of days. The same data set would have required a huge budget and a lot of time back in the 80s and 90s.

Today, you can collect data from hundreds of participants in just a matter of days. The same data set would have required a huge budget and a lot of time back in the 80s and 90s.

Though over the years, the method of execution has changed somewhat, we still focus on the same core objectives. We watch what people do, hear what their sentiments are, and gain a greater understanding of their attitudes and actions.

Assessing the Competitor

When you talk about the tools that allow you to crowdsource participants, you’re talking about stuff like, right? I know there are tools that let you inject some JavaScript into your website to record behavior behind the scenes. Others will randomly website visitors whether they’d like to participate in a survey.

That’s right, User Testing is an example, as is Userzoom.

With new technology, we are now able to assess the usability of websites of competitors. In other words, you can test websites that you don’t have control over. You can also easily assess applications that are being used on another user’s desktop.

Let’s talk about assessing applications offered by competing companies.  What are some tools you could recommend for analyzing a competitor’s SaaS product, such as a CRM or an ERP solution?

There are a couple of things I can recommend.

Previously, I worked for Oracle. At that time, you had client server apps which were much more complicated to analyze from a usability perspective. Now with the cloud and web-based applications, it’s much easier to record what people are doing. It’s easier now to watch participants complete tasks end-to-end.

The biggest barrier to assessing a competitor SaaS product is just getting through the authentication process. Luckily, there are several ways to do this. You can recruit participants who have accounts or who have the ability to test the product, say if they’re in the market for a CRM solution. They have to login to these sites and/or go through authentication steps, but you’re actually still able to record the screens as people go through and create a newsletter, set up sales contact information, etc.

With usability testing, it’s common to ask people to complete a core task on say three products: yours and two competitors. (It can be the same person or different people completing the task on different products.) Each time, you measure how long it takes to complete the task and the level of difficulty they report through standardized measures. In many cases, you’re able to record the screen that’s in the browser. With that, you have a record of what the participants are looking at.

Basically, this type of usability testing gives you a competitor benchmark.

You mentioned measuring how long it takes a user to complete a task. I imagine it was easier to measure that when participants were flown out to complete the tasks in a usability lab. How is that measurement taken these days, when usability testing happens remotely?

I understand how the qualitative perspective would be taken; you’re recording the screen and you’re watching their challenges. But if they’re using a competitor’s software, are you just qualitatively tabulating the amount of time between the tasks?

Usability Metrics

That comes from looking at efficiency metrics.  That said, there are three key metrics to keep in mind. The three core metrics are:

  • Effectiveness, the ability to complete the core tasks.
  • Efficiency, how long it takes to complete the core tasks.
  • Satisfaction, how users feel about the experience of using the product or service.

You need to have some objective success criteria when you talk about effectiveness. Often, we look for an artifact of whether the task was completed properly. For example, a confirmation screen may have a unique URL. There may be software that can see when users successfully reach this URL. In other cases, we simply ask them: “What was the confirmation message you got after completing this task?”. Again, usability tests are often unmoderated, in which case, we wouldn’t actually be watching them. So one measure of effectiveness is having the participant providing evidence that they have completed the task.

As for efficiency, as soon as we give them a task, and they tell us they’re starting, it’s really easy to start collecting time automatically. From that, we get a sense of how long it takes them to complete the task. You can get the duration per task and the length of time they spend completing the set of tasks.

To measure satisfaction, immediately after they say they’ve finished a task, we ask them how confident they were that they had completed it. For this, we have a standardized measure called the “single ease question.” We ask how easy or difficult it was to complete that task. We then add the response to a database and compare it to other user experiences.

Usability testing really gets some great insights. You may look at a user experience and see that it only took them 30 seconds to complete a task but their attitude about that experience was below average in relation to hundreds of other tasks collected in the database.

Info for the database is collected from survey responses, duration of tasks, and confirmations of having completed the task. Some of this data obviously has to be verified before it is added to the database. In other cases, automating that data collection is appropriate.

Finding Participants for B2B Usability Studies

Tell me about the recruitment process, because it’s obviously different for a B2B audience than for a B2C audience.

For B2C, you have a very large population to pull from and the ability to quickly build up samples that give you a lot of statistical significance.

That’s not the case with B2B, especially in niche markets. I imagine that recruiting is a little more challenging for B2B.

Yeah, absolutely. A lot of successful recruiting for B2B usability studies depends on finding the right partners who are able to provide and source those participants for you.

For example, right now, we’re doing a combination usability market research study for a large B2B tech company. We need to talk to the people who make decisions about backing up data. This is definitely not a general population. To reach these ideal participants, we have to be precise and creative about how we recruit.

Good points. If you look at some testing platforms, say where they ask you to do a 5-second test on a webpage, your unemployed college student would fit the bill perfectly.

…with complex B2B focused software or services, you need someone who actually has industry insight to participate in the study.

However, with complex B2B focused software or services, you need someone who actually has industry insight to participate in the study. You can’t test effectively if your participants wouldn’t actually use the system or do the tasks you’re studying. B2B companies don’t typically target novice, industry outsiders. Their target users are knowledgeable.

Right. Exactly.

What To Watch Out For

What are some challenges of conducting usability studies?

Getting the technology right is a big one.

It’s important that your study participants are able to log in successfully to whatever application you’re asking them to use.  You would be surprised how often this is an issue. You’re probably going to have to talk to someone in the provisioning department and make sure that is set up.

The account also needs to be properly set up after they’ve logged in. You don’t want to mar your study with an atypical experience that requires bizarre setup screens, etc.

Getting the accounts provisioned and properly set up is very important for usability studies, especially when examining a competitive space. Measuring the user experience of a competitor product or service requires the participant to have access to it. You’ll have to think out whether the participant is accessing the product or service through a free trial, a temporary subscription, physical access, etc.

It’s also important to make sure that you are asking participants to complete realistic tasks.

One particular challenge for B2B is that software platforms and services are often heavily customized by each enterprise customer to fit into a particular workflow. Payroll may be set up on one platform, meanwhile HR uses the same platform but has customized it so that it looks very different from what payroll is using. Move over to sales, and they’ve integrated the platform into Salesforce and other accounting software. In each situation, the platform is used differently. Bring all of these people into the lab and, if you’re not careful, you could be asking them to use software that they’re completely unfamiliar with. That doesn’t help your study at all.

Make sure the task you are giving participants is something they normally would do in their role. Otherwise, their user experience is rather irrelevant.

Make sure the task you are giving participants is something they normally would do in their role. Otherwise, their user experience is rather irrelevant.

Usually, you only have an hour or so with each participant. That means that the participants have to adapt to the conditions of the study quite quickly. They have to swiftly get comfortable with the fact that their screens are being watched. You may be across the country or physically sitting in a lab with an eye tracker on, and/or with cameras going.

Participants have to get used to being observed. That has to be factored into the data too. That participants may be taking time to adjust to the study conditions rather than diving right into the task at hand.

What’s next for usability testing?

What do you think is in store for the future of usability testing? What techniques or methods are on the cusp of being accepted into new common practice?

I think we’re going to continue to see more and better refined technological approaches for B2B usability studies. So much of software is now browser-based and can be accessed remotely. It’s going to be easier and easier to observe people using highly specialized systems without physically being there with them, without having to set up some customized tower in our lab that has the software built into it. That’s going to make testing easier. The methods are going to continue to get better.

Right now, B2B is still very much in a time-warp.

The B2B purchaser and user are often separated by a big space and time. The purchase of a B2B solution may be decided on by a team years out, before it gets configured and used. As a consequence, there are frequent basic usability issues that don’t get addressed until much later. That’s why we typically see 2-10 times as many usability issues in B2B as you see for a consumer-focused website or piece of software. You can see why usability testing is really important for B2B.

How Easy Is It To Use You?

Despite inevitable technological improvements, it’s still important for usability studies to focus on the basics:

  • Who are the users?
  • What are they trying to do?

Then watch them do it. Watch simple things like form fields and how they’re aligned and set up. Watch the labels and navigation. Find out how easy it is to use your product or service, where your pain points are, and how your usability compares to the competition. Time and technology won’t change those basic questions of usability studies or their importance to B2B companies.

This transcript was modified for clarity and readability by Isabel Gautschi.

This interview is brought to you by Cascade Insights. We specialize in market research and competitive intelligence for B2B technology companies. Our focus allows us to deliver detailed insights that generalist firms simply can’t match. Got a B2B tech sector question? We can help.

Image used courtesy of enotmaks/Fotolia

Home » B2B Usability Testing » Know Thyself: B2B Usability Testing
Share this entry

Get in Touch

"*" indicates required fields

Cascade Insights will never share your information with third parties. View our privacy policy.
This field is for validation purposes and should be left unchanged.