User Experience for a Better World
Eye Insights: How Many Participants do you Really Need?
How many participants do you need? This is one of the most frequent questions that I get when asked about eye-tracking. I find that the best way to explain it is to present a series of heatmaps generated from eye-tracking.
Let's look at a random sample of 3 different groups each composed of 8 people:
As you can see, there is a lot of variation from one heatmap to the next.
How about another 3 groups, this time composed of a sample of 15 people in each:
Well, there are now definitely some common areas where people look, but there are still some differences.
Finally, lets look at 3 more groups with a sample of 30 in each:
Now we start to see more consistent results from heatmap to heatmap.
So does this mean that we need at least 30 participants for every study? The answer depends on what you want to be able to say and do with the findings. As you start to reduce the number of participants below 30, you will not be able to make concrete predictive statements about your users' viewing behavior (i.e. everyone will look here, but will not look over here).
You can still learn from data collected from 15-30 participants but you need to be very careful in how that data is interpreted. You will also need to be careful in how you report findings from using a smaller sample size.
I'd like to hear your thoughts, experiences, and questions about sample sizes used with eye-tracking.
Let's look at a random sample of 3 different groups each composed of 8 people:
As you can see, there is a lot of variation from one heatmap to the next.
How about another 3 groups, this time composed of a sample of 15 people in each:
Well, there are now definitely some common areas where people look, but there are still some differences.
Finally, lets look at 3 more groups with a sample of 30 in each:
Now we start to see more consistent results from heatmap to heatmap.
So does this mean that we need at least 30 participants for every study? The answer depends on what you want to be able to say and do with the findings. As you start to reduce the number of participants below 30, you will not be able to make concrete predictive statements about your users' viewing behavior (i.e. everyone will look here, but will not look over here).
You can still learn from data collected from 15-30 participants but you need to be very careful in how that data is interpreted. You will also need to be careful in how you report findings from using a smaller sample size.
I'd like to hear your thoughts, experiences, and questions about sample sizes used with eye-tracking.
Views: 52
Comment
-
Comment by Jon Ward on June 6, 2010 at 5:57am -
Hi, as eye tracking specialists there are a few things that really need clarifying the above heat maps before anyone jumps to any conclusions. Firstly we need to be aware of the participants used during this testing, as their internet ability, knowledge of the site and persona and their interest or emotional tie to the site or task in hand. Secondly we need to put the heat map data in context, what was their task and goals in this testing - it appears that they have to find a link on the top page but were they required to do anything else? Thirdly we don't know what data this heat map shows, is it fixation based, time based or relative time based? Without the legend in the corner it is impossible to tell. The implications of this are huge - as a fixation based heatmap shouldn't be used for drawing conclusions, simply highlighting places where peoples attention was gathered. For web testing you should use a relative duration heatmap as this normalises the data across participants visits - and compensates for users that spend very little time, or a long time on the page. An absolute duration heat map should only be used when each participant sees the stimuli for the same amount of time, else a bias will appear for users that spent longer on task. This is also true of fixation based heat maps. The type of methodology used during testing is important too - as a combination of my previous points, plus say using concurrant think aloud will again alter the results as people fixate more, and for longer when trying to verbalise their actions.
Heat maps are a good visual for highlighting behaviour but to jump to any conclusions from them can be dangerous - especially when the points above aren't taken into consideration.
-
Comment by Suresh Susarla on June 3, 2009 at 4:25pm -
Andrew,
Good explanation on the size of participants There is also a statistical way of detemining the size of the study you want to conduct, which is called Power Analysis.
-Suresh
Videos
© 2012 Created by Diane Chojnowski.
You need to be a member of HFI Connect to add comments!
Join HFI Connect