![RSS icon [ Subscribe to Alex Horstmann's RSS feed ]](http://blobfisk.com/images/feed-icon-28x28.png){kind=icon}
BlobFisk.comMixing eye tracking and qualitative user testing
Eye tracking heatmap of the Channel 4 homepage
A couple of weeks ago we carried out some Eye Tacking testing on the Channel 4 website and I posted our initial findings soon afterwards. I’d now like to talk a little about the methodology that we used.
In traditional eye tracking studies the participant is presented with a page (in the case of web testing), given a period of time to look at the page, generally 15 seconds, during which the places that their eyes look is captured. This provides some very valuable feedback in that it shows where users look first, in what order they look at things and how long they look at things for.
However, it is merely a capture of the first look, and not linked to any task. People generally come to a website for a reason – they have a goal. When we carried out our eye tracking, we blended it with qualitative testing also. We carried out the initial gaze capture, but also tacked eye movement during specific task completion.
Why mix eye tracking with task analysis?
Heatmap of the Shameless page on Channel 4
We felt that to get the most out of our testing we would carry out some traditional eye tracking on some pages, but also capture gaze plots (and aggregate them into heatmaps) of task completion also.
By analysing the gazeplots and heatmaps that we got from task analysis we could understand better any issues in task completion, and using the initial eye tracking data, appropriately position information on the page based on the important tasks that users wanted to complete.
They are also very useful when looked at in parallel with our web statistics. Why users are clicking in certain areas and not in others is hard to work out from statistics alone, but with the results of our eye tracking these sort of questions are very quickly answered.
Some of the results
Gazeplot of the homepage showing numbered fixation points
We’re still going through the data ourselves, but I thought I’d give you a little peak at the type of raw information that we get back. For example, below a video of a live gaze plot of the homepage, and on the right is the static image of the same page with numbered fixation points (the fixation point is where the eye stopped to look and the number is the order in which those fixation points were looked at).
I’m really excited by the rich data that we’ve got from this exercise. Happily it’s confirmed that most of the decisions that we made in the design process were good ones – but it’s also highlighted some interesting issues here and there that we are already working to fix. For example, the interaction of the Catch-Up and Spotlight pop out drawer areas on the homepage.
Users see these areas (or at least the notice the faces, even though they are faded), but they don’t instantly grasp that these areas can be clicked on to reveal more information. To fix this we are going to make these drawers expand, smoothly, when a user hovers their mouse over the area for a certain period of time. This retains the click paradigm as bringing you to a new page or page state.
So, will I always mix eye tracking with qualitative testing from now on? I don’t think it’s as clear cut as that. I think it depends at what stage during the development of a page/site that you carry out the study. Earlier in the process, eye tracking in its traditional form is completely valid for testing on static designs to make sure important elements are not missed. But I think that later in the development cycle, and after release, mixing the two provides immensely powerful and userful inforamtion.
Tags: Channel 4, testing, Usability, User Experience, ux
This entry was posted on Sunday, February 8th, 2009 at 20:23 and is filed under Channel 4, Usability, User Experience. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
February 12th, 2009 at 10:35
It is interesting that you are showing the same phenomena that David Ogilvy (of the Advertising Agency fame) showed in 1970′s applies to the web as well as print, that peoples gaze is attracted to faces, and then drops down to look for a caption. Poynter did some high n (90 participants) eye tracking studies that showed the same phenomena as well for Newspaper sites. What would be interesting is if people didn’t look. It would be interesting to see if David Ogilvy other rules for Print Advertising applied to the web as well.
On the case of people hovering over the bottom bar, it may be worth looking at Ed Chi’s (Zerox PARC) Information Scent Theory. The theory builds upon Herb Simons (Nobel Laureate and father of Behavioural economics) theories of Satisficing.
The Information Scent concept is that the user is hesitant in clicking, becasue of the time cost of the click. The brain is deciding if I click that cost 12 seconds or do a keep on hovering that takes 1 second, will bring them closer to there goal. Most behaviour on the web is goal based (hence red routes). The channel 4 website takes about 3 seconds to load. Therefore cost of a click is about 10 -12 seconds, the user clicks, waits for the new page to load, decides that this is not what they are looking for, clicks the back button, and then waits for the page to load again. Because of this time penalty in clicking the user is hesitant in clicking.
The longer that a page takes to load the more hesitant a user will be on clicking further pages. This is a an argument against flash.
A hover probably takes less than two seconds. The more information that you can show when the mouse hovers over, the more confident that the user would be that their click is the right one.
February 13th, 2009 at 11:00
Hi James,
Thanks for your comment and for the really interesting references.
I could not agree with you more about Flash and the really adverse effect it has on page load times. I’m not completely anti-Flash, I think it has valuable uses in audio and video on the web, but I don’t think that it is the right technology to use for websites.
The reference you make to Information Scent is an excellent reason for this, and I will also add accessibility and usability into the mix. Accessibility in that, generally, most Flash implementations are not accessible. This can be put down to the developers not making it so, but it is extra work and the technology is inherently not accessible.
Usability wise it beaks many of the common ways that people use websites – and the one example I will point to is the back button. I think that the figure of 1/3 of all clicks online are the back button is still reasonably valid. In most cases Flash interfaces will not respect this (again, it is something that can be made to happen but it’s extra hoops to jump through).
The concept of red routes is really growing on me, and it is a very powerful arguement for your case for showing as much as possible (without sacrificing readability and usability) on mouse hover.
Cheers!
Alex
February 16th, 2009 at 17:42
I 100% agree with you on flash.
If the user goes to a page and then immediately clicks on the back button is it becasue they are lost?
I was thinking more when the user hovers using some of the ideas at http://www.dontclick.it/ The user hovers and the bar expands.
James
February 17th, 2009 at 07:36
SEO is a Contact Sport | Play to Win!» Blog Archive » Eye Tracking is Eye Catching says:[...] Mixing eye tracking and qualitative user testing | BlobFisk.com [...]
March 2nd, 2009 at 18:09
Hi Alex,
Was wondering if there was an update to this post? We employ a similar methodology – combining qualitative usability testing with eye tracking – and was interested to see what you came up with.
Really interesting post.
March 2nd, 2009 at 18:12
Hi Jennifer,
Thanks for your comment. I do plan to write a follow up, I just need to find the time! We’ve done the analysis of the testing but I just need to write the blog post on how we got on, what we found etc.
I’ll bump it up my To-Do list – thanks for the poke!
March 6th, 2009 at 12:02
Results of the eye tracking on Channel4.com | BlobFisk.com says:[...] to Jennifer for prompting me to write this post. Of all the user testing I’ve done with Channel4, this [...]
March 6th, 2009 at 12:03
Ok – that post is now published:
http://blobfisk.com/results-of-the-eye-tracking-on-channel4com/
March 19th, 2009 at 12:56
I also wanted to link to a Usability News article on this very topic:
http://www.surl.org/usabilitynews/72/eyetracking.asp
Summary:
This article discusses how eye-tracking can be used to supplement traditional usability test measures. User performance on two usability tasks with three e-commerce websites is described. Results show that eye-tracking data can be used to better understand how users initiate a search for a targeted link or web object. Frequency, duration and order of visual attention to Areas of Interest (AOIs) in particular are informative as supplemental information to standard usability testing in understanding user expectations and making design recommendations.
December 27th, 2009 at 10:35
Guardami! « zoomart – blog – magazine says:[...] Per ulteriori informazioni sull’argomento: ‘You look where they look’, ‘Mixing eye tracking and qualitative user testing’, ‘10 Useful Usability Findings and Guidelines’ [...]