BlobFisk.com

There is no better way of informing decisions around design, usability and user interaction with a web page than testing it on real users. The results that you get from testing are second to none, and have made a huge difference to the latest versions of the Channel 4 website, which I’ve had the pleasure of being involved with.

Unfortunately, Formal testing is expensive and time consuming. You need to pay for the use of a testing lab; pay for recruitment; spend time preparing the recruitment screener and test script; spend 2 days testing (with, ideally, 2 people taking notes); and write up your results and recommendations. The results are great but the cost is high.

There is an alternative though! Guerilla testing.

The Formal Way

The process I use when conducting formal, lab based user testing sessions is this:

1.  Prepare the recruitment screener based on the site’s Personas,
2.  Write the test script (this can take a bit of time!),
3.  Test the script internally to see how long the test takes, tweak questions etc.,
4.  Conduct the tests. These are usually run over 2 days, using 10 participants. The test lasts an hour and we take the users through a number of tasks on the site, getting their thoughts and feedback throughout. 1 or 2 people are in the observation room taking detailed notes.
5.  Review the notes, with the recording of the sessions and summarise, with recommendations.

This is quite a time consuming process, and can take up to 2 weeks. In projects with a tight deadline and even tighter budgets this sort of testing can be too much.

The Guerilla Way

Jakob Neilson tells us that testing with 5 people will find 85% of usability issues, and testing with 15 will find close on 100% of issues (Alertbox, 2000: Why You Only Need to Test With 5 Users). This is very interesting. It means that if you can quickly and easily test your site/application with 5 users you will find the vast majority of issues, and you if you do this early enough, you can fix them.

The next test, again with 5 users, you are testing the fixes you made on the back of the first test and find the majority of the remaining issues. Interestingly, Neilson says that this second test allows you to:

probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs. These important issues are often obscured in initial studies where the users are stumped by stupid surface-level usability problems that prevent them from really digging into the site.

This methodology is possible using formal, lab based testing. However, this is expensive. Instead, you can conduct guerilla testing. Ad-hoc, on the fly testing. You still need to recruit participants that match your Personas, but you carry out the test in a meeting room, canteen, coffee shop… How though?

I’ve been looking at Silverback, a guerilla testing application from Clearleft. It’s lightweight and very cost effective ($50). The downside is that it only runs on a Mac, but buying the hardware and software is still cheaper that a formal lab session, and it’s reusable.

It is slightly harder if you want note takers in guerilla testing, as they need to be within earshot of the participant. However, Silverback uses the built in camera (or external web cam) to record the user, captures what the user is seeing on the screen and shows where the user has clicked. The person running the test can insert mark points to act as a way point for questions or when something interesting has happened.

I really like the guerilla testing method. It’s fast and effective, and allows you to get very quick results. I don’t think that it will become a replacement for formal lab testing, but I see it becoming a more prevalent asset in the User Experience and Usability professional’s toolbox.

There are other, remote, testing solutions coming to the fore, like Userfly and Webnographer, which have the potential to provide great insight. However, nothing beats the act of watching someone interact with your site/software – that’s when you get the real insight into what is good and bad and what needs changing.

Tags: testing, Usability, User Experience

This entry was posted on Wednesday, January 21st, 2009 at 14:01 and is filed under Usability. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.