Top usability testing mistakes and how to avoid them

We’ll help you avoid usability testing mistakes

Our user researchers and usability experts have been in this field for 15+ years, and we are happy to share our experiences.

Not every mistake appears every time, of course, and there are also other ways to mess up your tests. The top usability mistakes we are listing here represent the most common usability testing oversights that can also significantly affect the overall goal of the testing, which is to improve the product or digital service.

The usability testing process requires a lot of preparation and dedication. To help you with it, we’ve put together a longer post with eight common usability testing mistakes we’ve seen and are sharing tips on avoiding them.

• Not choosing meaningful scenarios to test
• The wrong persona tested
• Not getting the most from the user and making assumptions
• Not understanding what the outcome of tests represents
• Testing for nothing
• Not having all the paperwork in place
• Not thinking about the dissemination of results
• Conducting only one usability test

Not choosing meaningful scenarios to test — one of the key usability mistakes

The first question we ask ourselves when planning user tests is: What exactly do we test?

We can’t just go through the site and keep asking users, “do you know what this button does?” or “do you know what this is for?” because there’s no context around that question, and whatever the answer we get, it’s not valuable.

So we need a context, a story, that will make users get to that button on their own.

And that story represents the scenario that you are testing.

An example is “Using the XYZ service, find the nearest Mexican restaurant and make a reservation for tomorrow night.”

Choosing a scenario that isn’t frequently used is an obvious mistake. But the less obvious mistake is setting a too broad or vague scenario so that you can cover more of the interface and functionalities.

The scenario should be defined so that when the user hears it, it makes sense and neither seems incomplete nor overwhelming to memorize. That’s when the user can do things naturally, and the feedback has the greatest value.

The wrong persona tested

When doing the test with the user, it is important to get the person that would have normally used your product. And this is not because they will know how to perform tasks while others won’t.

It’s because they have certain expectations of the product, while those who usually do not need your product will only evaluate what they see.

For example, let’s say you are testing a banking system and want to see if users can learn about the bank’s housing loans.

Suppose you are testing this with a person who either already has a loan or has already planned to get one. In that case, they will be interested in tools like calculators (e.g., I say how much I earn and spend on food, clothing, etc., and it tells me what kind of loan should be ideal for me), simulations, or something like that. Someone who has never seriously thought about getting a bank loan might simply go through the scenario and not think about what might be missing.

You should carefully look at the people you bring to the tests to get the best possible feedback.

Not getting the most from the user and making assumptions

This is about how you facilitate the test. The most common mistake is assuming you know why someone said or did something and simply moving from there.

If the user has paused for longer than you would expect or has done something unusual, don’t assume you know why that happened and then ask, “Did you do this because…”.

Never ask closed questions (those that have YES or NO answers). You should even be careful about asking a question like “Why did you click that button?” because it kind of implies that it was a mistake, and users might start thinking they did something wrong.

Simply remind them to share your thoughts with you, and guide them to getting clarification of what they are currently doing or expecting to happen. You should never give the users any indication of whether their actions are the right thing to do. They should figure that out on their own.

You should only gather as much information and thoughts as possible without interfering with their thinking.

The most common mistake is assuming you know why someone said or did something and simply moving from there.

It usually happens that after you have already run the same scenario with several users, and many of them have had a problem with a certain part when another user is having a problem there, you might assume it’s for the same reason. Don’t do that. Approach every user fully and objectively without being influenced by previous tests. Keep an open mind!

Not understanding what the outcome of tests represents

Usability tests are good for finding problems, not solutions! That cannot be stressed enough.

The usability testing results cannot tell you which solution would be optimal. It only tells you (if you have performed it correctly) where there is a problem and why that is a problem. To know which solution is optimal (out of a few), you must test them and see which one performs the best.

Sometimes users themselves have ideas about how they would solve a problem they just encountered.

Of course, it does not mean you should do what they said, but don’t dismiss their idea. If anything, it shows you what they are thinking. It’s just one person. It’s not a representative sample, but it gives you valuable insight into their reasoning.

Testing for nothing

We’ve seen this too many times and had to mention it here.

The clients do everything they need to test with the users. They get the recommendation on what should be changed and how, and then simply put that aside for now because of some other development priorities. Usually, this ends up with about a third of the problems being solved eventually, and the rest of them never.

True, the explanation of why to test with users only says to identify usability problems, which is what the tests do, but that’s not where your job stops. The tests are just the first part of the whole initiative, and the end goal is to have a better product, not just be aware of the problems in your current one.

When planning for the tests, plan for the development effort afterward. Have that in mind when scheduling and defining your budget to avoid this and similar usability testing mistakes.

Sure, some of the problems the tests will identify might be so fundamental that they are not easy to fix. That’s understandable. You don’t have to fix all of them. Those that are show-stoppers, sure, but not all of them.

However, what we are talking about here is the mindset that we’ve picked up from some clients after the tests, where they seem to be happy with finally understanding the problems but not keen on solving them in some conceivable future.

Not having all the paperwork in place

If you are recording testing sessions, planning on using direct users’ feedback, or sharing information and data from testing, you must obtain informed consent from users. This is particularly important if you are residing in or doing tests with citizens of the European Union.

You must adhere to GDPR rules, meaning that every user involved in your testing has the right to get the data you collected, ask for corrections, remove the data, and withdraw their consent even during the testing.

To avoid unexpected situations, make sure you have the paperwork in place before the actual session starts. In many cases, you might want to ask them to sign an NDA because you might involve them in testing a new product or a service that is not yet available on the market or otherwise needs to be protected. So, at minimum, you need to have a signed informed consent and an NDA and ensure it’s all aligned with GDPR.

Of course, checking your local regulations and ensuring you are aligned with them is wise, and you’ll be sure to avoid usability testing mistakes related to paperwork.

Not thinking about the dissemination of results

A usability test may yield a wealth of data, but if the individuals responsible for design decisions are unaware of what has happened, the test is a failure.

Also, all findings must reach the design and product teams. And a typical way of achieving that is by writing usability audit reports. The general idea is to summarize key findings and recommendations and comprehensively understand the issue.

Those reports can then also be shared with clients, who’ll be able to get an understanding of the study, findings, and recommendations.

But there’s a problem with reports. Most of them are never read, and a lot of times, writing a report is more difficult than it seems. It’s a complex project that spans different disciplines.

Our experience shows that reports are useful; clients typically love them and find them interesting.

However, we achieve better results if we deliver a follow-up workshop and open discussion about our usability findings with the client. It’s frank, direct, unscripted, and often reveals interesting insights from everyone involved. Those review sessions are now our primary method of delivering usability testing results, with reports being a support and a document that can be shared easily among the team.

Conducting only one usability test

It’s been said earlier in this post that usability testing is great for recognizing problems but fails when identifying solutions. Most experienced teams will not have a shortage of ideas on approaching the specific usability issue detected. Sure, some of those solutions might not be technically or financially feasible, and some might be too complex to sustain, but generally, there’ll be ideas and solutions.

But how to know which solution to implement after the first pilot test?

Use an iterative approach. Do an A/B test and see how the product or service works with the implemented solution. No one expects you to find a perfect solution right off the bat. It’s all part of the testing and never a waste of time. Go back to the design phase if need be.

Don’t be afraid to rely on data and tests. At least one follow-up round of tests after the proposed solution implementation should happen, and ideally, even more.

Let’s chat

Are you interested in being a part of an experienced team that works on world-class products? Jump to our Careers page and apply. We will help you grow your development, engineering, and soft skills.

Or maybe you have a great idea for a product and want to make it a reality? If you are unsure what the best next steps you need to take to deliver user-friendly products, reach out to us to schedule a free consultation with our experts. We’ll guide you through the whole process.