Abstract
We analysed the eye-tracking data of 147 participants as they used a total of 15 separate website navigation menus to complete key activities. The hypotheses for this study were that (a) the psychological phenomenon of the order effect would manifest in that items at either end of a menu would be located more quickly than those in the middle and (b) that the items that were relevant to completing the user’s tasks would be located more quickly through peripheral visual identification of these items. Although items relevant to the user’s task were acquired 1.8 seconds faster on average, both of the hypotheses were rejected as no statistically significant patterns were found. It was concluded that each user was likely to have his or her own searching behaviour and this could be affected by other factors such as the graphic design of the menu.
Practitioner’s Take Away
This study makes available an interpretation of a large set of eye-tracking data that many practitioners may not have access to and allows some interesting take away messages in the use of eye-tracking technology in usability studies:
- When designing a web menu, or advising on its design, do not spend much time deciding the order of menu items on the basis that it will help users find the items more quickly.
- Continue to adhere to cultural expectations such as the “Home” link always being on the far left.
- When conducting eye-tracking studies, use consistent project setups where possible so that data can be compared across projects. This allows such meta-analysis as performed in this study to further identify patterns in eye tracking and identify their causes.
Introduction
The order effect is a psychological phenomenon that has been observed since the beginning of the century. It describes a bias in the interpretation of a list of information depending on the order of presentation of that information. An early study on the subject found that content at the beginning of the list was learned better than that in the middle; dubbed the primacy effect, this is thought to be because items seen first are paid more attention and are observed by a fresh mind that has not processed any similar items yet (Deese & Kaufman, 1957). Additionally, the recency effect noted by Murdock (1962) describes items at the end of a list being recalled more accurately as they are still fresh in the reader’s mind.
The order effect has evolved from the context of “learning” to that of “finding” information, for example, there has been an assumption in the fields of advertising and marketing that placing the most important facts at the beginning and end of a message will optimise its effectiveness amongst potential customers. It is also said to have impacted on political elections, accounting for an increase in votes for candidates at the top of the list (Alvarez, Sinclair, & Hasen, 2006).
The consideration of the order of items is also very much relevant to the design of software, websites, and mobile applications in terms of the architecture of the user interface and placing the most important functions where they will be located the most efficiently for the end user. With the increasing emergence and maturity of usability for exploring the ease of use of software, it is relevant to question the extent of the order effect in this domain. Looking specifically at websites, it is useful to know if indeed there is a significant impact on the navigability of the site depending on the order in which menu items are presented to the user.
This study used an empirical approach to explore the time taken for web users to acquire website navigation menu items by tracking their eye fixations in these areas. The aim was to establish if the placement of a link on a menu—its distance from the middle—affected the average time it took participants to discover it. The hypotheses were that (a) the psychological phenomenon of the order effect would manifest in that items at either end of a menu would be located more quickly than those in the middle and (b) that the items that were relevant to completing the participants’ tasks would be located more quickly through peripheral visual identification of these items
Prior Work
Work by Yu and Roh (2002) examined the effect of different menu designs on an otherwise unchanged website, but did not examine the order of menu items. They found that the user’s searching and browsing performance was affected by the type of menu and that the drop-down menu style yielded better searching performance than a global and local navigation menu or a simple selection menu. However, browsing was found to be fastest when using the global and local navigation menu.
Various studies have addressed the effect of breadth and depth of menu structure on speed and accuracy of task completion, for example, Kiger (1984; as cited in Zaphiris, 1997) experimented with five menu designs with varying breadths and depths of links and found that using a menu of four items each of which contained a sub-menu of 16 items showed the fastest response time with fewest errors. However, in these studies, the participants completed tasks on multiple menus with the same link names in a rotating order, so there may have been some learning effect as to which link name to click on. Additionally, the order of items within each menu was not addressed.
There was very little empirical work that gives guidelines as to the interpretation of eye-tracking data. The overall application of eye tracking to usability is explored in “Using Eye Tracking to Compare Web Page Designs” (Bojko, 2006). This paper used number of gaze fixations as a metric of efficiency to explain usability differences between web page designs and commended the reliability of eye-tracking data and its ability to examine the users’ information searching process rather than just the outcomes of such searches.
The only available study incorporating order effect and eye tracking was made by user experience consultants at Etre, who used eye tracking to examine heatmaps for five popular websites in 2006. They noticed “some evidence of ‘order effects’ in the processing of the left-hand navigation menu” (Etre, 2006, Our Findings section, para 5) on one of these sites. However, insufficient information was provided on their methodology, such as the number and type of participants in the experiment and what the context of looking at the web page was. Additionally, the heatmap was not classified as to whether it represents the number of fixations on menu items or the duration of these fixations. This highlights the need to pursue further research in this area.
Method
A meta-analysis was made of several website usability projects over the past year at OTOinsights. A total of 15 unique navigation menus were included in the study, across 13 unique websites, with eye-tracking data from a total of 147 unique participants. Each menu was the primary navigation menu on the website, and data was taken from first exposure to this menu on the homepage (at the beginning of the usability study). The menu items were all textual and different numbers of words made up the menu item, meaning that some menu items occupied slightly more or less screen space than others. Examples of two such menus are shown in Figure 1.
Figure 1. Examples of the web menus that were analysed, a 9 item horizontal menu and a 10 item vertical menu
Each project followed the same methodology of a task-based approach with retrospective stimulated think aloud (RSTA) protocols. Participants were recruited from the public, meeting a demographic specific to the test websites (e.g., for a travel site, have booked a holiday in the past 6 months or intend to in the next 6 months). The participants were then invited to complete a series of simple tasks representing real world journeys they would complete on the websites (e.g., on a travel site, to make a booking for a holiday). Eye tracking was used in the first of these tasks to record the user’s fixation and saccade patterns across the web page upon their first exposure to it. The tasks were presented to the user on a sheet of paper for them to read aloud, the moderator asked them to clarify the purpose of the task, then the participant was told to begin as soon as the website had loaded. During the task both the moderator and participant were silent. Fixations represent moments when the eye stops moving and the brain directs attention to the area within the foveal field, i.e., the area the user is looking directly at. Saccades are moments of very rapid eye movement between fixations, where visual resolution is lost. Using the RSTA method involves allowing participants to complete eye tracked portions of usability studies without any interjections or requirements to think aloud, as these would influence their eye movements. After the given task was completed, the eye-track video was replayed to the user and they were asked to explain retrospectively what they were thinking, what they were looking for, and what their reactions to the stimulus at the time they completed the task were. The equipment used was a Tobii T60 infra red monitor that requires very little calibration, is non-intrusive, and accurate to 0.5 degrees.
During analysis, each web page stimulus was marked with Areas of Interest (AOI) that precisely enclosed each navigational menu area. The AOI enclosed the textual label used for the menu item, where tabs or buttons were used in the menu, the entire tab, or button. The Tobii software was used to calculate the Time To First Fixation (TTFF), a measure of how many seconds elapsed between task start and fixation inside an AOI. TTFF was chosen as a metric because it specifically addressed the aim of this study to establish if the placement of a link in a menu affected the average time it took users to discover it. There were several other measures yielded by the eye-tracking software, including number of fixations and duration of each fixation on a menu element, but these were considered out of scope as they occurred after the menu item was located, and this study was considering only the process of locating menu items, i.e., data before the menu items were located. TTFF values refer to the first time the user fixated on the menu item; they had never looked at it previously in the entire test session.
The TTFF data was amalgamated and marked up with the link name. We noted whether the menu was horizontal or vertical and whether the link was relevant to the task at hand. We calculated the distance of an item from the centre of the menu. This distance was calculated using the following formula:
Distance = (N/2) + 0.5 – P
Where N is the total number of items in the menu, and P is the position of the item in question. For example, an item that appears second in a menu of eight items the formula is the following:
D = (8/2) + 0.5 – 2 = 2.5
This formula was constructed specially for this test and was felt to be the simplest way to show numerically how far from the centre of a menu each item was. The number of menu items was halved to find the middle point, then 0.5 was added because the middle point was a whole menu item (e.g., the middle point of a 10 item menu would be in between items 5 and 6, or 5.5). Whether the item was at the beginning or the end of the menu was considered irrelevant for the first part of analysis, only its distance from the centre was considered. However, possible effects of position were considered in the results section.
Graphs were plotted for the distance from centre of the menu versus the TTFF for each website, and estimated for possible significant patterns. A graph of the mean TTFF for each unique distance was plotted and subjected to a Spearman’s rank correlation test to check for significance across the entire data set.
Results
A Spearman’s rank correlation test was conducted on the data to assess the relationship between the distance of a menu item (based on the above calculation) to its TTFF. The yielded p value was 0.178 showing little correlation. This implied that, for the menus tested, there was no significant impact of the distance of menu items from the centre of the menu on the time taken for users to direct their attention to them.
In practice, this means that users of websites were not consistently able to find menu items any quicker, regardless of whether they were at the start, middle, or end of the menu. This means that when designing a menu, less effort should be placed on the right position of items in the menu and more devoted to other factors that might include the terminology, taxonomy, and aesthetics, for example.
The results were cross examined to find any possible influencing factors.
Orientation of Menu
Three vertical menus were included in the study. Separate analysis of their data from the horizontal menus showed no significant effect on the TTFF from the distance of links from the centre of the menu (Spearman’s rank, vertical p = 0.505, horizontal p = 0.067). It must also be noted that one of the vertical menus was partially obscured by the page fold until users scrolled down; this would naturally increase the time taken to fixate on these menu items.
Practitioners should note that designing a vertical or horizontal menu does not seem to impact how quickly users can locate the desired item within the menus, although vertical menus run the risk of requiring page scrolling which does slow down navigation.
Size of Menu
The menus included in the study contained between 6 and 10 items; these were analysed separately and although they did show different patterns, none of them showed the expected pattern of TTFF increasing as distance from the centre of the menu decreases.
Most websites tend to have 6 to 10 primary navigation options, so the implications of this finding for website design are simply to state that designers should not panic that having a few more items will make it less efficient for users to find the one they want. For menus of more than 10 items, psychology research implies that users will have a harder time sifting through the information, but this is an area where a similar study could be performed in the future to assess performance on long menus.
Relevance of Menu Items on Task at Hand
Perhaps it is possible that users could more quickly recognise key words that are relevant to their current task; peripheral vision may be able to recognise familiar shapes or forms of words without directly placing the gaze on them or reading them. The eye-tracking technology does not record peripheral vision, only foveal vision, that is, the point on which the user is actively fixating. It was therefore interesting to classify menu items as to whether or not they were relevant to solving the task being completed at the time of eye tracking to see if there could be any impact of peripheral vision on helping users to spot relevant items more quickly.
Each menu item was classified as being either useful to the user’s task at the time the eye-tracking data was collected, or not (the path to the information requested in the task or what was thought to be the most direct path if there were multiple routes). The mean time to first fixation for items that were useful to the user was 10 s versus 11.8 s for items that were not useful, showing a slight increase in performance of locating useful items. However, when TTFF was plotted against the link’s distance to the centre of the menu, the expected decrease of TTFF with distance from the centre of the menu was not observed (Figure 2).
The Spearman’s rank correlation also showed no significant relationship in either case: p = 0.372 for items useful to the task and p = 0.154 for items not relevant to the task being completed. It must be noted that some users were able to complete their task without clicking on any menu items as relevant links were found elsewhere on the page, in which case fixations on the menu were coincidental to the task.
Figure 2. The time to first fixation varies with the distance (number of items) of menu items from the centre of the menu. Users were slightly faster at locating items that were relevant to their task, but there was no significant difference overall to items that were not related to the task at hand. Also, the error bars show a great variance of TTFF between each website menu.
Once again we did not see any significant impact of task relevance to TTFF; this was one possible explanation for why TTFF would vary that has been ruled out and forced us to consider other reasons why users would take longer to find the item they want. I suggest that this time was influenced by the aesthetics of the menu, including items around the menu which may draw attention; the user’s level of previous experience with other similar sites; and the users idiosyncratic search pattern. All of these are good topics for future research to help web designers make more effective menus.
Position of Item at Start or End of Menu
The primary aim of the study was to establish if a menu item’s distance from the middle affects the time it takes users to discover it, therefore top versus bottom positioning was outside the main scope. The initial formula described in the Method section of this paper does not differentiate between items located at the left and right sides of a horizontal menu or items at the top and bottom of a vertical menu. However such a discrimination was made by using the original formula and separating distances with a positive value (items at the start of a menu), with negative distances (items at the end of the menu). The average TTFF was then compared for each of these two groups.
It was found that items in the last half of the menu took 8 seconds longer to locate on average than those in the first half. However, the Spearman’s rank correlation did not show a significant relationship between the distance of the item from the centre of the menu and its TTFF for either items in the first half of the menu or the last half (p = 0.439 and p = 0.989, respectively).
Practitioners should note that there was no evidence in this study that more important items should go on the left and lead to progressively less important items on the right.
Other Measurements
The number of fixations before the target item was found was also considered. It may be the case that a long TTFF was produced by only a few fixations each of which lasted a long time or by very many shorter fixations. As shown in Figure 3, the mean average number of fixations before menu item acquisition varied from 10 to 40, but there was no pattern in items near the start of the menu taking more or less fixations to find. Once again, the error bars show a high degree of variability between each web menu in the number of fixations.
Figure 3. The average number of fixations before acquiring a menu item did not display a pattern when plotted against distance of the menu item from the centre of the menu. Error bars show the minimum and maximum number of fixations from the pool of web menus tested.
Recommendations
To progress this research further, it seems that further research could be conducted to identify if there are other factors apart from the order of menu times that may impact the TTFF, for example:
- Examine one menu system with many participants whilst varying the order of the items, and/or its graphical appearance, and measuring the TTFF. A large number of participants should be used to negate any outlying results and the design of the tasks should be such that they can only be completed using one of the menu items to ensure that users are not distracted by competing links.
- Consider assessing the TTFF in relation to cognitive style of the user.
- Study longer menus with more than 10 items.
- Measure alternative metrics to TTFF such as number of fixations and fixation duration.
Conclusion
There was no significant relationship between the distance of a menu item to the centre of the menu and the time elapsed before users first fixated on that menu item. There was a slightly increased performance when locating items in the first half of the menu as opposed to the second half. It’s interesting to speculate on what insights we would have if a significant relationship had been found: If users were usually faster at looking at and finding items at the beginning of a menu, it would be clear evidence for site designers to place all the most important items at the beginning of a menu and the less important items at the end. It could be argued that many designers follow this practice anyway, although this study has shown no evidence to show any benefit to that. Therefore, the implications on menu design are limited and show that less time should be dedicated to the position of menu elements and more time should be dedicated to the terminology, Information Architecture (taxonomy), and aesthetics of menus. I would argue the exception for conventions that users anticipate, such as “Home” being the first item and “help” and “My Account”/”Log in” links tending to be on the right hand side.
In conclusion, the location of the items does not have a great impact on how quickly users can find them (at least for horizontal and vertical website menus containing 6 to 10 items). The variance of location speed must be due to other factors, one of the likely candidates being the graphical treatment of the menu, which may attract attention to items based on their colour, shape, font used, etc. Another possible factor is the users’ conscious or subconscious searching patterns that they tend to follow when looking around a web page. It is important to remember that although no significant patterns were found overall, there were still many individual cases where some menu items took significantly longer to find than others, and each use of the site produced a non-generalisable pattern of behaviour. Studies should continue to examine the optimal order of menu items if it has been identified that important menu items are being located with less efficiency and more work is needed to consider longer menus with more than 10 items.
References
- Alvarez, R. M., Sinclair, B., Hasen, R. L. (2006). Election Law Journal: Rules, Politics, and Policy, 5(1): 40-56. doi:10.1089/elj.2006.5.40.
- Bojko, A. (2006). Using eye tracking to compare web page designs: A case study. Journal of Usability Studies, 3(1): 112-120.
- Deese. J., & Kaufman, R.A. (1957, September). Serial effects in recall of unorganized and sequentially organized verbal material. Journal of Experimental Psychology, 54(3), 180-7.
- Etre (2006, May 8). Five days: Dixons.co.uk. Retrieved on March 3, 2010, from http://www.etre.com/blog/2006/05/five_days_dixonscouk/
- Murdock, B.B. (1962). The serial position effect of free recall. Journal of Experimental Psychology, 64, 482-488.
- Yu, B. M., & Roh, S. Z. (2002). The effects of menu design on information-seeking performance and user’s attitude on the World Wide Web. Journal of the American Society for Information Science and Technology, 53 (11), 922-933.
- Zaphiris, P.G. (1997). Depth vs. breath in the arrangement of web links. Institute of Gerontology & Industrial and Manufacturing Department, Wayne State University, Detroit, Michigan.
