Five lessons I learnt from my first usability testing session

I have decided to start documenting my learnings, experiences and pretty much everything else on my journey to figuring out users, products and businesses.

I carried out my first usability testing session as part of a sprint task at work.

First, usability testing is a user research methodology in which the researcher observes users interacting with a product or prototype in order to evaluate its ease-of-use. The point is to see firsthand how effective the visual design, text and the general experience of the product is to the user.

Here are some of the things I learnt from the session:

Start with a pilot

A pilot is a small scale version of your usability test where you run through your script and the task scenarios with a small number of people. In my case, we did this with just one person (a new team member who wasn’t very familiar with the product). Doing the pilot helped us identify some areas of concern, figure out how to order tasks for the participants and forced us to think about how to handle scenarios that could occur during the session.

Have a backup plan

Users are crazy, they might mess up all your plans.

We reached out to 1000 people, 17 people signed up and a grand total of 0 people showed up 😭. This felt a little frustrating because my team had committed resources, time and effort to the session. A lesson I learnt from this was that we should have called the users a few days before the session to get verbal commitments from them (one of the users claimed not to have signed up for instance).

Thankfully, we had already planned to do some guerilla testing as well, so we spent all the time doing this instead. We walked up to random people at the co-working space that we were using (Apologies to the two women we accosted in the toilet 😂), briefly explained what we’re trying to do and asked them for a few minutes of their time.

This was our impromptu script for this:

“Hello,

My name is <insert name> and I’m from <insert company name>.

We are an <insert industry> company and we are trying to figure out how to make it really easy for people to <insert objective>.

We’re having a user testing session in <insert location>. Could you spare us a few minutes of your time? Yes, you can come whenever you’re free. We will be there till <insert time>

<Feel free to insert a cheeky remark such as “There’s small chops”>”

Quality over quantity

Of the 10 people that we spoke to, only 4 people showed up. That seems like a small number, but the sessions were really insightful as we were able to spend roughly 30 minutes with each person. By inviting them to meet us during their free time, we were able to get people who were actually interested in the session and willing to give us their time. We ended up getting lots of useful insight and feedback from watching them use the app.

Allow the user to make mistakes — resist the temptation to jump in to help or to direct them.

Even though I had read beforehand that researchers have to learn to stay away from “helping” participants, I was really tempted to “help” when I saw a participant stuck.

Luckily, I was able to catch myself and instead asked the user questions about what they were thinking. The real source of your insights, I think, will be in watching them struggle and figuring out why they are struggling. You’re not having the session to teach them how to use the product, you’re having it to learn how they use the product.

When you notice hesitations, instead of offering to help, ask the user “What are you thinking about?” or “What are trying to do right now”. They are likely to end up telling you what they find confusing in particular about the process.

Ask questions after actions and not before

Depending on your goals, you might need to ask the participants why they chose some options over others —ideally, this should happen after they have already made their decisions. You want to avoid hypothetical responses as much as possible so let them carry out the action and then ask them why they chose A and not B.

So “why did you do that?” instead of “why do you want to do that?”. If you ask them before, they might assume they are wrong and immediately try to defend themselves or do something else.

In all, we were pretty happy with the results of our study — we learnt a lot about the issues that newbies experienced with our app and started drawing up an action plan to fix the issues and make the experience better for them.

I look forward to doing more user research and trying out other methods.

📝 Read this story later in Journal.

👩‍💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.