The power of a usability study 

"Just push the button!" screams one of the engineers at the live stream of a user test, while my UX colleague, who is running the interview, keeps a patient poker face. The engineers discuss what they have just seen and ask us: "How can you be so patient, and not just help the user? They are obviously struggling."

…

"OK, they are struggling. We need to fix this!" is what one of the engineers said, and my work was done. OK, not really done, we still needed to fix it. But they understood that their perfectly implemented button did not work. Functionally-wise it of course worked, but users did not want to click the "Contact to get help on this topic" button (even though it was the only way out of this usability test). 

And this is how we iterated to a new design where users could selve serve their solution, instead of contacting the company. 

Why do a usability test? 

There is actually a whole list of reasons why you should do usability testing on a (new) design (or on your old design, to understand where the pain points are). Usability testing is NOT the same as quality assurance testing, and its goal is not to find bugs. Usability testing is about figuring out if the user is using your product as intended, and understanding the behaviour behind it. Running a test helps you:

• Find hidden pain points. Where do users get stuck, confused or drop off? You discover design flaws that were missed (and even the best designers need this).

• Save time and money. Honestly, how much time and money does a usability test cost compared to brand value loss, customer support tickets, and angry Reddit posts? If you fix problems early, they will not become a problem later.

• Prioritize features. If a usability test on a prototype shows you that users do not use your new feature, you can down-prio it. Or, just as important, you find out they wanted to use it but simply could not find it (hello, button!).

• Challenge assumptions. Every team has internal debates and assumptions about what is best for the user. Running a usability test shows you what users actually do, and gives you knowledge instead of questions.

How to do a usability test. 

"We do not have any fancy research lab." Not needed (but definitely nice to have).
"We have a hard time finding our users." That is a solvable issue (which I'll save for another blogpos).
"We do not have the time." That's why I am here.

There are two types of usability tests:

Moderated

You start by understanding what you want to learn, and build tasks around this goal. You write out the tasks (for example: "Play 5 rounds of this game and let us know when you think you are done"), and let the user finish task by task without interfering or helping them (that includes not laughing when they do something hilarious). You observe what happens, and after they have completed the tasks you can ask them why they made the decisions they made.

Remember the engineer screaming "just push the button"? This is why you don't. In real life there is nobody sitting next to your user to help them, so if you help, you learn nothing.

Tools you can use: A video call with screen sharing (Zoom, Google Meet or Teams) gets you very far (and is what I use 90% of the time). If you want more, Lookback is built for recording the screen and face at the same time, lets your team watch live, and helps you clip highlights afterwards.

Unmoderated

Well, that is a fake title, because you still need to moderate it. But you will not be able to step in live when the user goes off track. You create your task list and send it out to the user.
Important here is to ALWAYS TEST YOUR TEST with a user first. You can send it out to one participant and watch the result. Or run it live with a non-perfect user (which could be your neighbor), to see how they navigate through the tasks. You do not want to request 15 user videos, only to find out they are all useless because your tasks were not understandable.

Tools you can use: PlaytestCloud is great for games and mobile apps. They also have their own pool of players, so you do not have to find participants yourself. For websites and prototypes, Userinterviews, Maze, Lyssna or UserTesting are good options.

And then you can use:

Think out loud technique

You ask users to say everything they think while doing the tasks: "I am looking for the button… where would that be… maybe here?" This is great for understanding how people think and what they are looking for. The downside: talking while doing something is not natural, so people get slower, and sometimes more careful than they would be at home on their sofa.

Not think out loud technique

Users do the tasks in silence, just like in real life. This is important for seeing what people actually do and how much time it takes, which is great if you want to measure things like task time or compare two designs. The downside: you see what they do, but you have to ask afterwards why.

That's it! You can experiment with this yourself. Or, if you'd rather spend your time on building the product, I do it for you.

Show. don’t tell. And drive impact. 

I have learned a lot in my career. And one of the biggest lessons I learned the hard way: I told the engineers how much the users were struggling and how we had to fix it. That was the meeting where the engineers stopped liking me… No one likes to be told that their work isn't working, and they definitely do not want to be told how to improve their work.

We live and we learn. Depending on the team I am working with, I have 3 different ways of sharing insights to drive impact. What I pick depends on our working relationship and trust, and on the weight of the insights. (All-is-great does not need the same communication plan as your-6-month-investment-needs-a-full-redo.)

Snippet presentation and a workshop

Most useful for quick impact on complex systems, and for changing the roadmap in the short term.

I run a usability study and synthesise the results, finding opportunity areas that need improvement. Once I have a clear understanding of the problem, I start editing video snippets that show the observed problem (from different angles and different participants). I present the videos grouped by topic, together with the problem statement. No solutions. The team (anything from engineers and product managers to leadership) slowly builds understanding by watching different users. Now the important part of this delivery: a team workshop to discuss the results, in a safe environment where all ideas are welcome.
Because the TEAM comes up with the solutions, they are much more excited to implement them. They often come up with the same ideas I would have suggested, but the IKEA effect is a real thing. Because they understand why the solution is needed, and they feel ownership, the impact is way bigger.

Snippet video delivery and recommendations

Most useful for simple systems and direct impact, when trust is already established.

This delivery is basically the same as the previous one, but for when the system and insights are less complex, or when there is a lot of trust between the team and the researcher. This is never my first delivery with a new team, but after running a few usability tests together, I notice that the insight videos can be shorter and more powerful, and I can come with clear suggestions. This is often the case when an obvious change is needed (for example: users did not read the text, so we should add step-by-step icons).

Live observations and discussion

Most useful for long-term impact.

This is tricky. Time consuming. And so impactful in the long term! Highly recommended for team members who have never actually seen a real user in a neutral environment. Instead of me synthesising the results before the share-out, the team is welcome to watch the usability test live (via a hidden link). With most of my clients I always offer this option (feel free to drop in if you want!), but sometimes we organise it as the delivery itself. More on this in a future blogpost on the Empathy Power Hour.
Watching a full usability test together as a team is super powerful, because you see all the small interactions and get a better understanding of how people navigate through your product. This does not necessarily change your roadmap tomorrow (although it could, if we see obvious problems). But it does change your roadmap in the long term: everyone in the team builds their own picture of the user, and takes it with them into their own work. The engineer remembers that confused user while building the next feature, the PM while writing the next spec, and the designer while drawing the next flow.

Impact I have driven with usability tests

• Got onboarding for new users on the roadmap. After leadership saw first-time users struggling so much, they realised: this is not an easy fix. And it needs to be fixed.

• Made self-help available instead of contacting an agent. Remember the "Contact to get help on this topic" button? Our usability tests showed that some users really do not want to contact an agent. So we made self-help available, and as a positive side effect, it saved a lot of customer support contacts.

• Iterated on a new design that we thought was useful and super understandable, but turned out not to be understandable at all.

And the moments we skipped usability testing? Those resulted in long angry Reddit posts, an overflow of complaints to customer support, and rolled-back designs.

Want a usability test for your product? 

Can you run one yourself after reading this post? Technically, yes. Will it be easy? Well, probably not, but you will learn a lot! Think about writing tasks that don't steer the user, finding the right participants, keeping a poker face while someone ignores your perfect button, turning hours of video into clear insights, and then getting your team to actually act on them.

If you want to skip the trial and error, and get efficient impactful usability tests: I can help. I set up the test, find the participants, run the sessions, and share the insights in the way that will actually drive impact in your team. Or I train your team to do it themselves, with someone experienced next to them for the first few tests.

Send me a message and we can chat about how I can help!