Measuring the First Impression: Testing the Validity of the 5 Second Test

Abstract

This study aims at measuring the scientific validity of a method often used, but little formalized, to evaluate user experience: the 5 Second Test (5ST). To present this test, a fixed interface, such as a webpage or software screen, is shown to users for exactly 5 seconds. After 5 seconds, the interface disappears and test moderators ask a few open questions to elicit users’ first impressions.

To measure the validity of the method, an experimental protocol was developed to compare the elements that are most remembered when displayed for 5 seconds or for a period of time with no specified time limit. The hypothesis proposed is that the user will first focus on the elements specific to the non-instrumental qualities (hedonic qualities) of the interface during the first moments of the interaction, and then after 5 seconds, users can better perceive the instrumental qualities (practical qualities). The results partially validate this hypothesis and show, in my view, the interest in the 5 second test that complementary studies must continue to validate.

Keywords

5 second test, usability, aesthetic, perceived quality, user experience, evaluation method

Introduction

There are numerous methods to evaluate the usability of human-computer interaction (HCI), and more generally its user experience (UX). For example, the ISO 16982 (2002) selected a set of 12 basic methods, categorized by different criteria such as direct user participation or the phase of a system’s life-cycle. Similarly, in an article referencing the main methods that frame user centered design (UCD), Maguire (2001) distinguished 36 methods that he classified according to the main stages of the process. The literature supports many other references committed to the evaluation of HCI (e.g., Koutsabasis, Spyrou, & Darzentas, 2007; Stanton, Salmon, Walker, Baber, & Jenkins, 2006). After a review of the literature and a search of the major databases in HCI (ACM, ScienceDirect, IEEE Xplore, CiteSeer, etc.), I discovered that the 5 Second Test (5ST) was never referred to. However, the 5ST has become an increasingly widespread method used by usability practitioners to collect users’ first impression of an interface.

Even though I did not find references to 5ST in scientific research, there are several blogs that discuss the importance of using a 5ST to assess how effective websites are in conveying what their site is about, thus retaining the user’s attention for a longer period of time (Alabi, 2007; Perfetti, 2007). There are also 5ST tools available for people to use to assess their own website’s ability to convey its purpose in the first few moments a person is looking at their page (e.g., UsabilityHub: https://usabilityhub.com/five-second-test). When dealing with the cognitive ergonomics of HCI, the 5ST is being used as an assessment tool. Therefore, it is appropriate to question the usefulness of the 5ST method as a measure of a user’s first impression and to attempt to fill the lack of scientific validation of this method.

I devised this study from a general reflection on the theoretical foundations of the 5ST. There were two groups of participants: one group was asked to compare a screenshot of a webpage using the 5ST method, and the other group of participants were allowed to view the same screenshot without any specified time limit. I collected all participants’ first impressions. Participants gave their impressions of the website’s perceived quality, user experience, and perceived usability. The results are presented and discussed to lay the first scientific foundations of 5ST and its contribution to the measurement of an interface’s user experience.

The 5 Second Test

The following sections discuss the 5ST method in relation to its application protocol, theoretical contributions, and validity.

Application Protocol of the 5ST

The 5ST is a quick and inexpensive user research method that can be integrated into the design process very early and iteratively. The 5ST application protocol is simple and adaptable to a variety of user requirements. For Perfetti (2007) and Alabi (2007), the method first requires identifying the purpose of a page (e.g., a webpage or an application screen), and then establishing context and defining tasks that will be presented to users. The following is an example of a 5ST protocol:

1. The context of the page to be evaluated by users is presented (e.g., a commercial site, the screen of an expert software, etc.), as well as a short list of tasks.
2. Users are informed that a page will be presented to them for 5 seconds, and they should try to remember everything they saw during this short period.
3. The page displays for exactly 5 seconds. After 5 seconds, a visual mask or another screen with questions displays to cover the page.
4. Users are asked to describe everything they remember and how they could have achieved a certain task.

Other researchers do not describe the same protocol for 5ST. For example, some researchers present a fixed page without characterizing its context or specifying tasks. Once the page is displayed for 5 seconds, the user is simply asked to describe the objective of the page or what could be done with this page. For another example, researchers can choose to use tasks to guide the exploration of a page (Alabi, 2007; Perfetti, 2007) or allow users to explore the page by themselves. By using one or the other of these methodological protocols, it is possible to collect information on a user’s immediate understanding of a page and on its most prominent elements. These elements constitute the building of the first impression.

Theoretical Contribution to the 5ST Method: Forming a First Impression

The 5ST is based on the premise that a user can build an image and form a first impression of an interface’s contents in 5 seconds. This assumption seems to be accepted by cognitive ergonomists, but no theoretical basis has ever been advanced to explain or verify this assumption. Thus, I propose to position the 5ST in relation to the theoretical framework that explains the development of a first impression, starting with foundations emerging from social psychology, then comparing the framework to studies in HCI that mobilize similar concepts.

The first impression in social psychology

The first impression is a socio-cognitive process that is subjectively complex and has been studied for a long time in social psychology (Asch, 1946). This process represents how people perceive and process social interactions with others, including the very first moment of interaction. It is based on the first elements perceived through which we form a positive or negative judgment.

For the purposes of the theoretical framework for my research, I’ve summarized the process of forming a first impression by concentrating on the following five psychosocial interdependent paradigms:

• implicit personality theory (Beauvois, 1982; Schneider, 1973)
• impression formation of personality (Asch, 1946; Gollin, 1954; Hamilton & Sherman, 1996)
• halo effect (Clifford & Walster, 1973; Dion, Berscheid, & Walster, 1972; Lemay, Clark, & Greenberg, 2010; Thorndike, 1920)
• confirmation bias (Nickerson, 1998; Snyder & Swann, 1978)
• first impression formation (Fiske & Neuberg 1990)

Implicit personality theories (IPT) laid the foundations for socio-cognitive judgment processes of others. Beauvois (1982) and Schenider (1973) described how an individual builds expectations of another individual from a set of personality traits resulting from incomplete information. Beauvois (1982) pointed out that this is an inference activity because additional information relating to a person’s personality is inferred based on a limited exposure to that person. For example, a colleague described as dynamic, ambitious, and serious will be more readily considered reliable.

Impression formation is a continuation of implicit personality theory. In a series of now famous experiments, Asch (1946) submitted to different groups a set of adjectives meant to describe personality traits of an individual. The list of adjectives was identical from one group to the other, except for a single adjective. For example, in his first experiment, Asch sent the following six adjectives to two groups: intelligent, skillful, industrious, determined, practical, and cautious. In addition to these adjectives, one group received the word cold, and the other group received the word warm. Both groups were asked to describe in one paragraph the impression they had of a person that matched the list of adjectives. Then they were asked to assign a score (from 0 to 100) to another list of 18 adjectives according to the impression they had of this person. The results showed that the group to which the adjective warm was submitted judged the person as 91% generous, 90% happy, and 51% imaginative, while the group that had the adjective cold judged the person who they were assessing as 8% generous, 34% happy, and 19% imaginative. Asch’s research has shown that from a limited set of personality traits combined, individuals form a general and unified impression of others.

Bias judgment of others identified during the first impression was deepened by research on the halo effect (Clifford & Walster 1973; Dion et al., 1972; Lemay et al., 2010; Thorndike, 1920). The halo effect generalizes the first impression created by the physical appearance of a person by personality traits (Dion et al., 1972). Thus, a person considered physically attractive will be more readily considered to have a more fulfilling social life and most stimulating professional life compared to a person judged as unattractive. Clifford’s (1973) experiment with U.S. teachers illustrated this halo effect: After presenting a set of photographs of children to a panel of primary school teachers, they were asked to evaluate each child using several dimensions of the intelligence quotient (IQ) or each child’s chance of success at school. The results of this study showed that attractive children were assessed as more intelligent and having more chances of success at school than children deemed unattractive.

Confirmation bias is the tendency of an individual to seek or interpret any evidence in favor of his or her first impression (Nickerson, 1998). Thus, if the first impression is positive, the individual will tend to minimize the negative aspects of the surrounding elements and exaggerate the positive aspects. Conversely, the more a first impression is negative, the more the individual will tend to minimize the positive and accentuate the negative aspects.

Finally, the general process of first impression formation was described in its temporal dimension by Fiske and Neuberg (1990). They developed the continuum model of impression formation that applied to the perception of others. Briefly, this model is based on the principle of categorization: When an individual encounters a person, he or she immediately categorizes the person’s salient features, that is to say the central physical features of dress, smile, hair, and so on. Based on this categorization, the individual determines emotional, cognitive, or behavioral attributes to the person encountered. This judgment of others is achieved very quickly. Willis and Todorov (2006) estimated it takes about 100 milliseconds for individuals to form an opinion of others.

First impression in human-computer interaction

The process of forming a first impression previously described as psychosocial theories can be transposed to the formation of a first impression in human-computer interaction. Various authors have indeed found very similar phenomena.

Liu, White, and Dumais (2010) observed that the first moments of visiting a website were very critical as users’ loyalty to the site was conditioned in these first moments, that is to say their will to return to it or not. The authors described a behavior of „screen-and-glean“: users first explored the webpage very quickly in order to get a first idea of what the page was about, and then they decided on whether to stay on the page or not to search more precisely for information that might be of interest. Also, the first impression users formed of a webpage was decisive for their future interactions that directly affected loyalty and satisfaction with the site.

Just as the halo effect consists of relying on the physical characteristics of a person to infer personality traits, the same effect can be applied to the aesthetics of a user interface from which the first impression is forged and on which the user’s attention focuses. The works of Lindgaard, Fernandes, Dudek, and Brown (2006) have shown that the elements of a page that are most quickly analyzed were those related to graphics and visual aspects. Liu et al. (2010) have meanwhile found out that design elements (graphics) influenced the forming of the first impression the most because they attract the most attention. These are then followed by the elements relating to usability and credibility that are processed by the user. Tractinsky, Katz, and Ikar (2000) examined the impact of aesthetics on the perceived usability of a system. By adapting Dion et al.’s (1972) study title, „What Is Beautiful Is Good,“ Tractinsky et al. shaped their study to fit HCI—“What Is Beautiful Is Usable.“ They (and other researchers) found that an interface that had been judged aesthetically pleasing was perceived as easier-to-use than an interface that was considered unattractive. Aestheticism also promotes a greater user satisfaction with the system (Hassenzahl & Monk, 2010; Tractinsky et al., 2000).

Aesthetics are the reason why the design elements of an interface play a leading role in the construction of the user experience (UX; Hassenzahl & Tractinsky, 2006). According to Mahlke (2007), the UX can be considered as an integrative concept, including practical characteristics of an interface (instrumental qualities) and hedonic aspects (non-instrumental qualities). According to Thüring and Mahlke’s (2007) Component of User Experience Model (CUE-Model; see http://mecue.de/english/background.html for a graphical depiction), interaction characteristics, which are a function of system, user, and context, affect the perception of non-instrumental qualities, emotions, and perception of instrumental qualities which, in turn, have experiential consequences such as overall experience, acceptance, intention to use, and choice of alternatives.

Thüring and Mahlke (2007) distinguished instrumental and non-instrumental qualities: “instrumental qualities concern the experienced support the system provides and the ease of its use […]. Non-instrumental qualities, on the other hand, concern the look and feel of the system. Features such as visual aesthetics or haptic quality belong to this class. Hence, while instrumental qualities are closely related to the usability and usefulness of a system, non-instrumental qualities result from its appeal and attractiveness. The perception of both types of qualities is likely to influence the third component of user experience, i.e., the emotions that accompany the interaction process” (p. 263).

Thus, when users draw a first impression by focusing on the aesthetic characteristics of a webpage or a system, they will retain and process the non-instrumental elements.

Finally, in accordance with the confirmation bias, the more positive the first impression of a site is, the more users will tend to evaluate the entire site positively. Raita and Oulasvirta (2011) showed that expectations regarding the usability of a website based upon the first impression or upon reading an external evaluation of the site, conditioned the usability measured by users. In other words, the more users expect a site to be easy-to-use, the more positively they assess the usability of the site in relation to users who have a negative expectation. Kim and Fesenmaier (2008) demonstrated that the credibility of a site, which defines its persuasiveness, was determined during the forming of a first impression. Michalco, Simonsen, and Hornbæk (2015) showed the link between expectations and user experience: They primed users with a negative or positive short game description, then the users played a game and rated it based on several measures. The results of this experimental study showed that priming users influenced the following five measures related to UX: game rating, beauty, goodness, pragmatic quality, and hedonic quality.

A user’s initial impression of a website is formed quicker than the ability to discern how that website is useful to them. Lindgaard et al. (2006) showed that users could form an initial impression of a website in only 50 milliseconds, in the same way as if the site were presented to them for 500 ms. Also, Dahal (2011) discovered that if a webpage required 2.66 seconds to attract the attention of users, their first impression had already been developed after just 180 ms.

Research Objectives: Scientific Validation of the 5ST as a Tool for Measuring the First Impression in HCI

To compensate for the lack of scientific validation of the 5ST and to measure its importance as a method of measuring the first impression of an interface, I wanted to explore the specifics of the first impression of participants using the 5ST, and then compare the 5ST to a free exploration of a webpage without a specified time limit. In accordance with the literature, I hypothesized that the 5ST would allow participants to form a first impression more focused on the design and aesthetic elements of the page, in comparison to an unlimited time exploration. In other words, I expected that the participant’s attention would be more focused on non-instrumental qualities of the interface (aesthetics, attractiveness) during the first moments of interaction and more focused on instrumental qualities (utility, usability) after 5 seconds. See Figure 1 for a schematic representation of my model.

Figure 1. Schematic representation of the construction of a first impression and my research problem concerning the user perception of instrumental and non-instrumental qualities of a system according to the temporal course of the interaction.

Methodology

The National Meteorological Service in Luxembourg engaged the Luxembourg Institute of Science and Technology to conduct a usability study on their new version of their website: meteolux.lu. As a Senior Researcher for the institute, I developed this study and analyzed the data. I gathered the data with the help of Adélaïde Chauvet, a student researcher.

This website was chosen because of its informative nature, easy comprehensibility, and low visual load. This site provides information on the Luxembourg meteorology: current weather, five-day forecasts, and weather warnings.

The following sections discuss the experiment design and procedures, the scales and questionnaires, and the participants who took part in this study.

Experiment Design and Procedure

Two experimental conditions have been compared for this study:

• 5SEC condition: For this condition, the cover page of a website (meteolux.lu; Figure 2) was displayed for exactly 5 seconds. After 5 seconds, a visual mask was displayed to cover the page.
• Unlimited time or UT condition: For this condition the same page was displayed until participants clicked a button labelled „Give us your feedback“ when they thought they had taken enough time to explore the webpage.

The independent variable in this study relates to the viewing duration of the cover page of the meteolux.lu website. The displayed page was a screenshot (a static element), without any possibility to interact (Figure 2).

Figure 2. Website presented to users for 5 seconds (5SEC condition) or for an unlimited time (UT condition). The page was a screenshot, non-interactive.

Both configurations were developed in JavaScript and tested on multiple computers to ensure a maximum compatibility with the major browsers on Mac OS and Windows (Chrome, Firefox, Internet Explorer, and Safari) and to ensure, in particular, that the first page displayed for exactly 5 seconds in the 5SEC condition.

Both experimental conditions were presented to two participant groups, with random assignment of participants to each condition. Table 1 shows the experiment instruction given to the participants for each condition.

Table 1. Instructions Provided to the Study’s Participants

5SEC condition	UT condition
For the first step, the page of a website will be presented for 5 seconds. Explore it carefully. Then, for the second step, you will be asked several questions to collect your impression with regard to this webpage. We will ask you to answer as honestly as possible. This won’t take more than 5 minutes. During the display of the page, please stay focused, and do not try to interact with it (for example by clicking on a button, buttons are not active). When you feel ready, click on the button „Start the study“ below.	For the first step, the page of a website will be presented. Explore it carefully. Then, for the second step, you will be asked several questions to collect your impression with regard to this webpage. We will ask you to answer as honestly as possible. This won’t take more than 5 minutes. During the display of the page, please stay focused, and do not try to interact with it (for example by clicking on a button, buttons are not active). You can choose to respond to questions whenever you want by clicking the button „Give feedback.“ When you feel ready, click on the button „Start the study“ below.
Original French version: Dans un premier temps, la page d’un site web vous sera présentée pendant 5 secondes. Ensuite, dans un second temps, plusieurs questions vous seront posées pour recueillir vos impressions vis-à-vis de cette page. Nous vous demanderons d’y répondre le plus sincèrement possible. Cela ne vous prendra pas plus de 5 minutes. Pendant l’affichage de la page, restez concentré(e) sur celle-ci et n’essayez pas d’interagir avec la page (par exemple en cliquant sur un bouton : les boutons ne sont pas actifs). Quand vous vous sentez prêt(e), cliquez sur le bouton „Commencer l’étude“ ci-dessous.	Original French version: Dans un premier temps, la page d’un site web vous sera présentée. Explorez-la attentivement. Ensuite, dans un second temps, plusieurs questions vous seront posées pour recueillir vos impressions vis-à-vis de cette page web. Nous vous demanderons d’y répondre le plus sincèrement possible. Cela ne vous prendra pas plus de 5 minutes. Pendant l’affichage de la page, restez concentré(e) sur celle-ci et n’essayez pas d’interagir avec la page (par exemple en cliquant sur un bouton : les boutons ne sont pas actifs). Vous pourrez choisir de répondre aux questions concernant vos impressions quand vous le souhaiterez, en cliquant sur un bouton intitulé „Donner vos impressions“. Quand vous vous sentez prêt(e), cliquez sur le bouton „Commencer l’étude“ ci-dessous.

 

Having looked at the website, each group was asked to answer a set of questions, more specifically described in the following section. These questions focused on their overall impression of the perceived quality, the perceived usability, and user experience of the homepage.

Scales and Questionnaires

After their assigned exposure to the website image, participants were asked to complete the following scales and questionnaires:

• The first questionnaire consisted of two open questions and one closed question:
  • The identified objectives of the page: In your opinion, what is the goal of the presented website?
    (Original item in French: Selon vous, quel est l’objectif du site qui vous a été présenté ?)
  • The visual elements they remembered (menus, buttons, icons, advertising banner, etc.): What element of the webpage did you remember?
    (Original item in French: Quel(s) élément(s) de la page avez-vous retenu(s) ?)
  • The general impression they had of the page (very bad to very good on a Likert scale from 1 to 7): What general impression did this page give you?
    (Original item in French: Quelle impression générale cette page vous a-t-elle laissée ?)
• The 11-item quality scale, named NetQu@l and originally developed by Bressolles (2006; see Appendix A), included the dimensions of information quality, design, and usability. I did not include the other dimensions, reliability and security/confidentiality, from the original scale because they are used to evaluate commercial websites.
• The third scale measured perceived usability of the homepage with a 10-item System Usability Scale (SUS; Bangor, Kortum, & Miller, 2008) translated into French.
• The AttrakDiff scale, in its full version with 28 items (see Appendix B), was used to evaluate the user experience (Hassenzahl, Diefenbach, & Göritz, 2010). I used a French translation validated by Lallemand, Koenig, Gronier, and Martin (2015).
• A 7-point Likert scale was used to ask the participants for their age, their gender, and their degree of familiarity with technology (1 = Not comfortable with to 7 = Fully comfortable with).

Each of these scales and questionnaires allowed for measuring the non-instrumental or instrumental quality of the website according to the user’s perception and the CUE-Model (Thüring et al., 2007). See Table 2 for how the scales and questionnaires relate to the assessment of the interface qualities.

Table 2. Study Scale and Questionnaire Comparison to the Non-Instrumental or Instrumental Qualities of the Interface

Scales and Questionnaires		Non-instrumental qualities	Instrumental qualities
Identified objectives			X
Remembered visual elements		X
NetQu@l	Information quality		X
	Design	X
	Ease of use		X
SUS			X
AttrakDiff		X	X

 

Participants

The participants were recruited using a call via two French mailing lists: ErgoIHM and RISC Infos. ErgoIHM aims at diffusing information from the HCI field. The mailing list has a register of around 3,000 subscribers from many profiles and interests. RISC Infos distributes information from various disciplines of cognitive science: psychology, neuroscience, philosophy, artificial intelligence, connectionism, or linguistic. This mailing list includes people (from all professional backgrounds, age, and gender) who volunteer for research studies.

I registered 169 respondents for both of the conditions. I selected 146 participants after excluding those who already knew of the website or did not complete the questionnaire answers. The participants used their own computers (Mac OS or Windows) to take part in the experiment. They were asked to use one of the following browsers because I had successfully tested the conditions using these browsers: Chrome, Firefox, Internet Explorer, or Safari.

After a random allocation, the 5SEC group consisted of 68 participants (average age 38 years, 26 men and 42 women), and the UT group consisted of 78 participants (average age 40 years, 18 men and 60 women).

Study Results

The following sections discuss the methods used to process the results and the general results of the 5ST and UT tests in addition to the results of each questionnaire.

Methods and Result Processing

First and foremost, factor and reliability analyses were performed on the NetQu@l, SUS, and AttrakDiff scales. Cronbach’s alphas were 0.95 for AttrakDiff, 0.83 for the SUS, and the following for each of the NetQu@l dimensions: 0.91 for design, 0.89 for the information quality, and 0.94 for ease of use. The Cronbach’s alpha for the SUS, lower than Bangor et al. (2008) found with a score of 0.91, is probably due to the French translation that was not scientifically validated. Each of these scales has very good internal consistency, with an alpha above 0.70 according to Cronbach’s recommendation (1951), and had been validated for this study.

The statistical software SPSS 19 was used to process the results. Qualitative data (perceived objectives of the page and the remembered visual elements) were manually coded and processed with MS Excel to get a map of the most frequently recalled items (see Figures 3 and 4).

Concerning the identification of objectives (hereafter denoted as the dependent variable objectives), the following three main objectives of the site were identified: (a) actual weather in Luxembourg, (b) weather forecast for Luxembourg, and (c) weather alerts. The objectives scores had a value from 0 to 3, depending on the number of objectives identified by the participants.

Measurement of the First Impression in the 5ST or UT Condition

The following sections discuss the results in general, the results of each scale or questionnaire (objectives, AttrakDiff, SUS, and NetQu@l), and the perceived elements of the webpage.

General results

The synthesis of the results obtained by comparing the 5SEC and UT conditions are presented in Table 3.

Table 3. Averaged Comparison Score for the 5SEC Group^a and the UT Group^b

Scales	Groups	M	SD	t-test c
Objectives	5SEC	1.00 /3	0.24	t(126)=-0.91; p=0.37
	UT	1.05/3	0.42
AttrakDiff	5SEC	4.41/7	0.91	t(144)=-0.75; p=0.45
	UT	4.52/7	0.85
SUS	5SEC	71.87/100	14.94	t(144)=-1.75; p=0.08
	UT	76.25/100	15.18
NetQu@l	5SEC	4.87/7	1.14	t(144)=-2.24; p=0.03
	UT	5.27/7	1.00

^a n = 68. ^b n = 78. ^c significant differences in bold.

The results show that the 5ST is effective for measuring the level of objective identification because there is no significant difference between the two groups, t(126) = -0.91, p = 0.37. At the same time the user experience measured by AttrakDiff also does not present a significant difference, t(144) = -0.75, p = 0.45.

However, some differences are suggestive with respect to the SUS scores, t(144) = -1.75, p = 0.08 and NetQu@l, t(144) = -2.24, p = 0.03.

These results are described in detail in the following sections.

Objective identification

Table 4 shows the results of how the participant-identified objectives compare to the client-identified main objectives. Again, the three main objectives of the homepage defined by the client (The National Meteorological Service) were (a) the current weather, (b) five-day forecasts, and (c) weather warnings. Using an open question (In your opinion, what is the goal of the presented website?), I collected the participants’ perceived objective for each condition. Thus, I compared whether or not the declared objectives matched with the official objectives.

Table 4. Perceived Objectives Compared to the Official Objectives

	Official objectives of the meteolux.lu website
	Current weather		Weather warnings		Five-day forecasts
	Number of correct identification	%	Number of correct identification	%	Number of correct identification	%
5SEC (n=68)	66	97.1	2	2.9	0	0.0
UT (n=78)	72	92.3	7	9.0	2	2.6
Total	138	94.5	9	6.2	2	1.4

AttrakDiff scale

In its full version with 28 items, the AttrakDiff scale has four dimensions, with seven items by dimension. The results for each dimension are presented in Table 5.

Table 5. Averaged Comparison Score for Each Dimension of the AttrakDiff Scale

Dimensions of AttrakDiff	Group	M	SD	t-testa
Pragmatic Quality	5SEC	5.23	0.95	t(144) = -2.54, p = 0.01
	UT	5.64	1.00
Hedonic Quality—Identification	5SEC	4.53	0.94	t(144) = -0.34, p = 0.73
	UT	4.58	0.80
Hedonic Quality—Stimulation	5SEC	3.12	1.23	t(144) = 0.29, p = 0.77
	UT	3.06	1.24
Attractiveness	5SEC	4.75	1.24	t(144) = -0.36, p = 0.72
	UT	4.82	1.11

^a significant differences in bold

Only one dimension, the Pragmatic Quality, indicated a significant difference between the UT and 5SEC groups, t(144) = -2.54, p = 0.01.

NetQu@l scale

It was noticed that only the scores of NetQu@l’s design dimension did not differ in the two groups (Table 6).

Table 6. Averaged Comparison Score for the 5SEC and the UT Groups for NetQu@l

Dimensions of NetQu@l	Group	M	SD	t-testa
Information quality	5SEC	4.96	1.21	t(144) = -2.19, p = 0.03
	UT	5.36	0.96
Perceived ease of use	5SEC	5.03	1.20	t(144) = -2.77, p = 0.01
	UT	5.58	1.19
Design	5SEC	4.44	1.38	t(144) = -0.45, p = 0.65
	UT	4.54	1.29

^a significant differences in bold

Perceived elements of the webpage

Finally, I compared the elements that were most selected by users in both conditions. I graphed the results by plotting the percentage of recognition of each visual element. See Figure 3 for the 5SEC group, and see Figure 4 for the UT group.

Figure 3. 5SEC group: Representation of the recognition percentage for the webpage elements.

Figure 4. UT group: Representation of the recognition percentage for the webpage elements.

Elements concerning the country of Luxembourg and the meteorological icons were observed in both groups, but only the UT group visualized and remembered elements concerning the weather forecasts for the next five days. In addition, the 5SEC group noticed the blue color on the page (color of the menus, the titles, and the banner), but they could neither identify the elements’ form nor their content. The UT group meanwhile remembered that there was a red lion on the top left of the page.

Discussion: Forming a First Impression

Regarding the forming of a first impression, the results of this study show, first and foremost, that the 5ST is not enough time to allow users to identify all the objectives of a website. As this study showed, a significant number of participants in the 5SEC condition only identified the Current Weather objective, few identified the Weather Warning objective, and none of the 5SEC participants identified the Five-Day Forecast objective. Presumably, the participants’ first impression was built on the central and essential elements of the webpage.

In general, the results of my study show that the participants’ attention did not seem to focus systematically on the non-instrumental qualities (hedonic aspects) during the first moments of interaction and then focus on the instrumental qualities (practical characteristics) after 5 seconds, as I had previously expected (Figure 1). The attentional process does not seem to follow a single, linear way, but a multiple and complex way through the interaction time.

Indeed, I did not record significant differences for the perceived non-instrumental qualities during the UT and the 5SEC conditions. The assessment of the interface design, measured by the NetQu@l scale, showed no significant difference, t(144) = -0.45, p = 0.65 between the two conditions. This was the same result for the dimensions for hedonism and attractiveness on the AttrakDiff scale: t(144) = -0.34, p = 0.73 for Hedonic Quality– Identification; t(144) = 0.29, p = 0.77 for Hedonic Quality—Stimulation; t(144) = -0.36, p = 0.72 for Attractiveness. Only the blue zone (Figure 3), which corresponded to a menu frame on the left of the webpage, reflected the participants’ perception of an aesthetic element. Therefore, it is not really possible to conclude, as I had previously formulated in this paper, that the user’s attention would be more focused on non-instrumental qualities of the interface (aesthetics, attractiveness) during the first moments of interaction.

On the other hand, it seems that the participants focused more on the instrumental qualities after 5 seconds when they browsed the webpage. Indeed, the scores of the SUS showed a trend to a better assessment of usability quality for the UT group than the 5SEC group, t(144) = -1.75, p = 0.08. Similarly, the dimensions of the NetQu@l relative to the instrumental qualities (Information quality: t(144) = -2.19, p = 0.03 and Perceived ease of use: t(144) = -2.77, p = 0.01) had a significantly higher score for the UT condition. Note also that the Pragmatic Quality of the AttrakDiff dimension seems to confirm this trend, with a significantly higher score for the UT group than the 5SEC group, t(144) = -2.54, p = 0.01. Finally, the elements of the webpage and the objectives perceived by the participants during the period of unlimited exploration better matched the defined objectives of the website (current weather, weather warnings, and five-day forecasts). This seems to show that the participants’ perception during the UT condition was more focused on the usefulness of the system, which is another essential element of the instrumental quality, in addition to the perceived usability.

This initial attention to UX and design elements could also be explained by the integration of attributes theory formulated by Treisman and Gelade (1980) in the field of cognitive psychology. The authors considered attention as a link that combines distinct and different elements (color, shape, orientation, etc.) of an object in a unique representation. According to the theory of the integration of attributes, we sometimes split our attention so that all the constituents of a stimulus (e.g., a webpage) are processed at the same time, and sometimes we focus on one element of the stimulus so that only one element is processed at a time. In addition, the following two forms of attention, shared and selective, are part of a continuum:

• The first stage is shared attention. It grants the automatic registration of attributes (for example the elements of a webpage) by using parallel processing where all the presented information in the stimulus is processed simultaneously, quickly, and with a „pop-out“ effect („eye-catching“ elements).
• The second stage of the theory is focused attention. It defends the idea of serial processing that requires the identification of objects that follow each other. The focused attention is a more demanding method of treatment that becomes necessary when objects are complex.

Regarding the 5ST, it seems that the participants’ attention for some of the time focused on general elements with attention to the eye-catching attributes, such as a sun or the blue color of the website that I tested. However, when it comes to processing information more specifically, the 5 second time limit imposed by the test was not sufficient for the participants to apply focused attention. Thus, complex objects, such as the weather forecasts chart for the next five days, cannot be considered from an intentional point of view. In other words, there seems to be a hierarchy of perceived elements on a webpage in the development of the first impression. First, the design’s graphical elements are explored, perceived, and thus retained. Then these elements help to create a user experience for hedonic aspects. And, finally, the elements that determine the ease of use (usability) are considered. In a study on the impact of aesthetics on the cognitive processes of website users, Bonnardel, Piolat, and Le Bigot (2011) showed that the dominant color of the site played a key role in the exploration strategy of the website and its subjective judgment by users.

Conclusion and Perspectives

This study explored how the 5ST method can be used to measure how users form their first impression using this time-specific method. This method, often mentioned in UX design blogs, has never been scientifically proven or even explained from a theoretical point of view. I sought to bring some first ideas to be considered for the scientific validation of this method, which nevertheless needs to be further pursued. Indeed, researchers need to be careful to not generalize these findings. The small sample for each condition is not a negligible limitation. It would be useful to apply this study to a bigger sample, around 300 people for each condition, in order to analyze the data in greater detail. Moreover, using a meteorological website, such as the one used in this study, is very specific and may surprise some users. It would be interesting to test the 5ST with a commercial website or a social network, which were the most popular website categories in 2016 (Alexa, 2016).

Future studies should examine why usability is not evaluated in the same way in 5 seconds as when browsing a page without a time limit. However, using a 5ST seems to be sufficient time to gather the user experience of an interface and the quality of the design elements. The paths in which I progressed concerning the theory of the integration of attributes seem interesting to me, but would require further study.

Similarly, an additional study using other media (e.g., websites, computer systems, or mobile applications) seems essential to compensate for the unique case of this study. I plan to measure the impact of a mobile application icon (available on the major platforms—Google Play or App Store) when forming a user’s first impression. In addition, I plan to use an eye-tracking system to gather more concrete data about the most explored elements of an interface in less than 5 seconds.

Tips for Usability Practitioners

When assessing the first impression of an interface, practitioners should consider the following when using the 5 second test (5ST):

• The 5ST captures the very first moments of the human-computer interaction. These few moments will affect the entire period of interaction.
• The first impression is based on specific elements of the interface: color, icon, and more specifically, esthetic elements. So use the 5ST to test your graphical charter or your corporate identity guidelines.
• Keep in mind the 5ST must be applied to a static webpage or a high fidelity mockup.
• The usability aspects are probably perceived after 5 seconds so it is probably not a good idea to use the 5ST to check the usability problems of your interface.

Acknowledgements

I would like to acknowledge and thank Adélaïde Chauvet for her strong contribution to this study. I also thank Jim Lewis and the anonymous reviewers for their valuable comments and suggestions.

References

Alabi, A. (2007). Will your website pass the 5 second test? Retrieved May 15, 2016 http://ezinearticles.com/?Will-Your-Website-Pass-The-5-Second-Test?&id=409089.

Alexa (2016). The top 500 sites on the web. Retrieved October 21, 2016 from http://www.alexa.com/topsites.

Asch, S. E. (1946). Forming impressions of personality. Journal of Abnormal Psychology, 41(2), 258–290.

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594.

Beauvois, J. L. (1982). Théories implicites de la personnalité, évaluation et reproduction idéologique [Implicit theories of personality, evaluation and ideological reproduction]. L’Année Psychologique [The psychological year], 82(2), 513–536.

Bonnardel, N., Piolat, A., & Le Bigot, L. (2011). The impact of colour on website appeal and users’ cognitive processes. Displays, 32(2), 69–80.

Bressolles, G. (2006). La qualité de service électronique : NetQu@l Proposition d’une échelle de mesure appliquée aux sites marchands et effets modérateurs [The digital service quality: NetQu@l. Proposition of a measurement scale applied to commercial websites and moderator effects]. Recherche et Applications En Marketing [Research and Applications in Marketing], 21(3), 19–46.

Clifford, M., & Walster, E. (1973). The effect of physical attractiveness on teacher expectations. Sociology of Education, 46(2), 248–258.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.

Dahal, S. (2011). Eyes don’t lie: Understanding users‘ first impressions on website design using eye tracking. Master’s Thesis, Information Science and Technology, Missouri University of Science and Technology.

Dion, K., Berscheid, E., & Walster, E. (1972). What is beautiful is good. Journal of Personality and Social, 24(3), 285–290.

Fiske, S. T., & Neuberg, S. L. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. Advances in Experimental Psychology, 23(1), 1–74.

Gollin, E. S. (1954). Forming impressions of personality. Journal of Personality, 23(1), 65–76.

Hamilton, D. L., & Sherman, S. J. (1996). Perceiving persons and groups. Psychological Review, 103(2), 336–355.

Hassenzahl, M., Diefenbach, S., & Göritz, A. (2010). Needs, affect, and interactive products—Facets of user experience. Interacting with Computers, 22(5), 353–362.

Hassenzahl, M., & Monk, A. (2010). The inference of perceived usability from beauty. Human-Computer Interaction, 25(3), 235–260.

Hassenzahl, M., & Tractinsky, N. (2006). User experience—A research agenda. Behaviour & Information Technology, 25(2), 91–97.

ISO 16982. (2002). Ergonomics of human-system interaction—Usability methods supporting human-centred design.

Kim, H., & Fesenmaier, D. R. (2008). Persuasive design of destination web sites: An analysis of first impression. Journal of Travel Research, 47(1), 3–13.

Koutsabasis, P., Spyrou, T., & Darzentas, J. (2007). Evaluating usability evaluation methods: Criteria , method and a case study. Proceedings of the 12th international conference on Human-computer interaction: Interaction design and usability (pp. 569–578). Berlin Heidelberg, Germany: Springer-Verlag.

Lallemand, C., Koenig, V., Gronier, G., & Martin, R. (2015). Création et validation d’une version française du questionnaire AttrakDiff pour l’évaluation de l’expérience utilisateur des systèmes interactifs [A French version of the AttrakDiff scale: Translation and validation study of a user experience assessment tool]. Revue Européenne de Psychologie Appliquée/European Review of Applied Psychology, 65(5), 239–252.

Lemay, E. P., Clark, M. S., & Greenberg, A. (2010). What is beautiful is good because what is beautiful is desired: Physical attractiveness stereotyping as projection of interpersonal goals. Personality & Social Psychology Bulletin, 36(3), 339–353.

Lindgaard, G., Fernandes, G., Dudek, C., & Brown, J. (2006). Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour & Information Technology, 25(2), 115–126.

Liu, C., White, R. W., & Dumais, S. (2010). Understanding web browsing behaviors through Weibull analysis of dwell time. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval – SIGIR ’10 (pp. 379-386). New York, NY: ACM.

Maguire, M. (2001). Methods to support human-centred design. International Journal of Human-Computer Studies, 55(4), 587–634.

Mahlke, S. (2007). User experience of interaction with technical systems. Technische Universität Berlin.

Michalco, J., Simonsen, J. G., & Hornbæk, K. (2015). An exploration of the relation between expectations and user experience. International Journal of Human-Computer Interaction, 31(9), 603–617.

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.

Perfetti, C. (2007). 5-second tests: Measuring your site’s content pages. Retrieved May 15, 2015 from http://www.uie.com/articles/five_second_test/.

Raita, E., & Oulasvirta, A. (2011). Too good to be bad: Favorable product expectations boost subjective usability ratings. Interacting with Computers, 23(4), 363–371.

Schneider, D. (1973). Implicit personality theory: A review. Psychological Bulletin, 79(5), 294–309.

Snyder, M., & Swann, W. (1978). Hypothesis-testing processes in social interaction. Journal of Personality and Social, 36(11), 1202–1212.

Stanton, N. A., Salmon, P. M., Walker, G. H., Baber, C., & Jenkins, D. P. (2006). Human factors methods: A practical guide for engineering and design. Brookfield, VT: Ashgate Publishing Company.

Thorndike, E. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4(1), 25–29.

Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human–technology interaction. International Journal of Psychology, 42(4), 253–264.

Tractinsky, N., Katz, A., & Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13(2), 127–145.

Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.

Willis, J., & Todorov, A. (2006). First impressions. Making up your mind after a 100-Ms exposure to a face. Psychological Science, 17(7), 592–599.

Appendix A

Original and translated items used in this experiment from the NetQu@l questionnaire.

	Original items in French	Translated items in English
Information quality	Ce site fournit une information détaillée sur les services proposés	This site provides detailed information on the services offered.
	L’information sur ce site est pertinente	The information on this site is relevant.
	L’information sur ce site est précise	The information on this site is accurate.
Perceived ease of use	Ce site est facile à utiliser	This site is easy to use.
	Il est facile de chercher de l’information sur ce site	It is easy to search for information on this site.
	Il est facile de se déplacer et de trouver ce que l’on cherche sur ce site	It is easy to get around and find what to look for on this site.
	L’organisation et la mise en page de ce site facilitent la recherche d’informations	The organization and layout of this site facilitate the search for information.
	La mise en page de ce site est claire et simple.	The layout of this site is clear and simple.
Design	Ce site est jolie.	This site is pretty.
	Ce site fait preuve de créativité	This site is creative.
	Ce site est visuellement attirant	This site is visually attractive.

Appendix B

Original and translated items from the short version of the AttrakDiff questionnaire.

Dimensions	Original English version	Translated French version
Pragmatic Quality	Technical – Human	Technique – Humain
	Complicated – Simple	Compliqué – Simple
	Impractical – Practical	Pas pratique – Pratique
	Cumbersome – Straightforward	Fastidieux – Efficace
	Unpredictable – Predictable	Imprévisible – Prévisible
	Confusing – Clearly structured	Confus – Clair
	Unruly – Manageable	Incontrôlable – Maîtrisable
Hedonic Quality – Identification	Isolating – Connective	M’isole – Me sociabilise
	Unprofessional – Professional	Amateur – Professionnel
	Tacky – Stylish	De mauvais goût – De bon goût
	Cheap – Premium	Bas de gamme – Haut de gamme
	Alienating – Integrating	M’exclut – M’intègre
	Separates me – Brings me closer	Me sépare des autres – Me rapproche des autres
	Unpresentable – Presentable	Non présentable – Présentable
Hedonic Quality – Stimulation	Conventional – Inventive	Conventionnel – Original
	Unimaginative – Creative	Sans imagination – Créatif
	Cautious – Bold	Prudent – Audacieux
	Conservative – Innovative	Conservateur – Novateur
	Dull – Captivating	Ennuyeux – Captivant
	Undemanding – Challenging	Peu exigeant – Challenging
	Ordinary – Novel	Commun – Nouveau
Attractiveness	Unpleasant – Pleasant	Déplaisant – Plaisant
	Ugly – Attractive	Laid – Beau
	Disagreeable – Likeable	Désagréable – Agréable
	Rejecting – Inviting	Rebutant – Attirant
	Bad – Good	Mauvais – Bon
	Repelling – Appealing	Repoussant – Attrayant
	Discouraging – Motivating	Décourageant – Motivant

 

Published in: in Volume 12, Issue 1,