Evaluation of a Game Used to Teach Usability to Undergraduate Students in Computer Science

Abstract

There is a growing recognition of the importance of teaching usability, which has been discussed in various forms in undergraduate courses in areas related to Information Technology. Usability is an essential concept that professionals need to learn as they produce artifacts for different types of users and contexts. After a systematic mapping of the literature in this field, the authors found that the techniques currently used to teach concepts related to usability mostly involve the development of projects and case studies or the application of heuristic evaluations. Although the use of serious games has been used in different areas, our research exposed that serious games are not used to teach usability. Therefore, we propose in this paper the development of a simulator game that exposes the player to a corporate environment by simulating real situations in the projects of a fictitious company. The objective of the game, called the UsabilityGame, is to support the teaching of usability by addressing the usability engineering life cycle, requirements analysis, and prototyping and heuristic evaluation. We conducted four experiments to evaluate the effectiveness of using the UsabilityGame as a tool to teach usability concepts to students. We concluded that the game promotes the learning of usability in general, and heuristic evaluation and requirements analysis in particular.

Keywords

usability engineering, teaching strategies, learning strategies, serious games, educational games

Introduction

The andragogical model (teaching adults to learn) is different than the pedagogical model (teaching children to learn). In adult training, the emphasis is more on the process and less on the content and the instructor. Adults need to know why they have to learn something. They learn best in an experimental way when the learning is based on problem solving and the topic is of immediate value (Day, Harrison, & Halpin, 2009). For Drappa and Ludewig (2000), one of the keys to educational success is motivation, and one of the best motivations for learning comes from experience. Therefore, the challenge is to use a form of teaching—in the case of this research the teaching of human-computer interaction (HCI) and specifically usability concepts—in a concentrated and motivated way, providing an understanding of the key concepts and enabling the student to apply the HCI life cycle in practice.

Serious games, that is, a digital game that simulates real-word experiences with the goal to educate users rather than a goal to entertain users, has become an interesting tool in this andragogical context. Serious games can help teach practical concepts and help support the development of cognitive skills through the application of knowledge as it relates to the activities in the game. These computer-based games enable the simulation of real situations and provide immediate feedback that encourages students to learn by doing, by making mistakes, and by analyzing their decisions. Furthermore, the use of games in education has been driven mainly by the high motivational factor that this tool can bring to learning (Gibson, Aldrich, & Prensky, 2007).

A study by Connolly et al. (2012) examined the literature on computer games and serious games, with focus on the potential positive impacts of gaming and particularly with respect to the learning value and skill enhancement. Connolly et al. identified over 7,392 papers in their review of the literature between 1996 and 2009. The amount of papers confirmed the surge of interest in this area. However, at the time of this research, no serious games that were specifically designed to teach HCI and usability concepts were identified.

Teaching computing professionals HCI concepts is important and should be included as part of computer science courses at all levels (ACM & IEEE, 2008; ACM SIGCHI, 2009; Pow-Sang, Rusu, Zapata, & Roncagliolo, 2009; Rusu & Rusu, 2007). Usability is a core concept of HCI, and one of the most common definitions of usability that is appropriate for this concept is defined by ISO 9241-11 (2010): “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” Usability refers to a set of concepts such as time of execution, performance, user satisfaction, and learning facility (Abran, Khelifi, Suryn, & Seffah, 2003). Activities designed to promote usability are studied for usability engineering; these activities are carried out in the process of software development, which creates a connection between users and developers (ACM & IEEE, 2008; Cooke & Mings, 2005; Mayhew, 1999).

In this paper, we review the relevant literature involving teaching HCI, specifically what guidelines are used to teach HCI, how usability engineering relates to HCI, and how (or if) serious games are used to teach usability. We then describe the UsabilityGame software. This is followed by a discussion of the study design, a report of the results obtained for a study involving 47 students, the conclusions we discovered, and the empirical implications of the study for computer science educators.

Background and Related Work

In the following sections we present some guidelines on important aspects to be learned in the area of HCI. The aim of this guideline review is to support the choice of subjects and scenarios represented in the UsabilityGame. We then discuss a usability engineering approach, which we used to develop a script for the game. Finally, we provide background to the use of serious games, especially games that are used in computer courses, and present a systematic mapping of the methods and techniques used to teach HCI and usability as it relates to using a serious game to teach those concepts.

Guidelines for Teaching HCI

Since 1968, the Association for Computing Machinery (ACM) and the Institute of Electrical and Electronics Engineers (IEEE) have maintained the Computer Science Curriculum (CSC; 2008) that provides guidelines for content related to the field of Computer Science. One of the core foundations within the CSC is the goal to create a “process for user-centered development” with regards to HCI, meaning that a focus on how developers design products with the user in mind is important (ACM, IEEE 2008).

The ACM SIGCHI Curricula for Human-Computer Interaction is a report developed by the ACM Special Interest Group in Computer-Human Interaction (SIGCHI) to provide recommendations for education in HCI. For the Computer Science curriculum, the committee recommends that courses “focus on training a new generation of system designers, builders, and implementers who are truly concerned with those who will ultimately use their software” (ACM SIGCHI, 2009, p. 60). SIGCHI recommends that students have familiarity with a wide variety of relevant disciplines, software engineering among them.

Usability Engineering Life Cycle

Mayhew (1999) presented an approach to the usability engineering life cycle, focusing on users’ performance when carrying out their tasks and prioritizing design systems tailored for their characteristics and objectives. This approach provides a complete and detailed description of activities involved in each stage of a proposed cycle of usability engineering (Figure 1). We used this approach during the development of the UsabilityGame software.

Figure 1. The usability engineering life cycle. From The Usability Engineering Lifecycle, by D. J. Mayhew, 1999, San Diego, CA: Academic Press. Reprinted with permission.

The Requirements Analysis phase includes defining user profiling (understanding the users), task analysis, consideration of technical- and platform-related capabilities and constraints, and establishing general design principles; these components define the usability goals. The style guide that is produced out of this analysis documents the specifications for users.

The Design/Testing/Development phase has three levels. Level 1 is the conceptual model or design, a mock-up (low-fidelity prototype), that is evaluated to eliminate major design flaws. Level 2 consists of the screen design standards, designing the basic interaction styles on the screens and then evaluating the designs; this sub-phase allows for testing against usability goals. Level 3 is the detailed interface design, a fine-grained design of the interface and interaction, that allows further testing against the usability goals.

The purpose of the Installation phase is to provide input for the maintenance and enhancements phases of the product life cycle design and development.

Teaching Usability and Serious Games

Serious games usually refer to games used for training, advertising, simulation, or education that are designed to run on personal computers or video game consoles (Susi, Johannesson, & Backlund 2007). Simulation-style games are a powerful educational tool that has been successfully used in various fields because they allow for situations to be introduced and practiced, experiments to be repeated, and alternatives to be explored, all of which enable students to gain a deeper insight to each specific situation (Navarro, 2006).

Connolly et al. (2012) carried out a systematic literature review of the empirical evidence on computer games, and in particular serious games, in regard to the potential positive impacts of gaming for users aged 14 years or above, especially with respect to learning, skills enhancement, and engagement. The findings revealed that playing computer games is linked to a range of perceptual, cognitive, behavioral, affective, and motivational impacts and outcomes. The most frequently occurring outcomes and impacts were knowledge acquisition, content understanding, and affective and motivational outcomes.

Games are currently available for teaching content related to different areas of computer science, such as computer fundamentals (Sindre, Jahre, & Natvig 2009), programming (Muratet, Torguet, Viallet, & Jessel, 2011), and operating systems (Hill, Ray, Blair, & Carver, 2003). In particular, when considering the use of games in software engineering, there are several games that have been developed to support the teaching of different content in this area. A systematic review on the subject of games used for software engineering education (Wangenheim & Shull 2009) found 12 different games, with computer-based simulations dominating the list of games being used for education. A breakdown of the studies by subject matter and learning domains reveals that most were developed to teach software project management knowledge.

Do games exist for HCI? How are aspects related to usability taught? The following section seeks to answer these questions.

Teaching Usability: A Brief Mapping

To determine if the literature provided an answer to our questions of how usability and HCI are taught, we completed a systematic mapping study. This type of study is designed to provide a general overview of an area of research, seeking to establish whether any research evidence exists on a topic, and to provide an indication of the amount of evidence if it is available (Kitchenham & Charters, 2003). In Sommariva and Benitti (2012), the process of systematic mapping is detailed. In this paper we only report on the main studies that relate to our theme:

• Carroll and Rosson’s (2006) paper discussed the use of a case study where students performed all the steps related to the life cycle of usability.
• Tao (2005) discussed the application of heuristic evaluation to projects developed by the students themselves.
• Wahl (2000) proposed a method where students were separated into groups and asked to develop a library automation system. Each group was then asked to evaluate the usability of the software developed by the other group.
• Faulkner and Culwin (2001) used students to conduct a usability study involving 124 users who used a set of web pages and answered questions about the usability of those pages.
• Ludi (2005) proposed a method that used a hands-on approach that enabled students to apply various techniques to address usability in a system. The students followed a usability testing process that gave them the opportunity to plan the process and methodology, recruit participants, conduct the testing, and analyze the test results.

It is important to note that these studies did not present any findings related to our proposed approach that we present in this paper. This indicates a need for empirical research related to teaching HCI. Another point to highlight is that despite the extensive use of games for teaching computing (as discussed in the Teaching Usability and Serious Games section), we did not find any studies that relate research to the development and use of games for teaching usability. In view of this gap, this paper presents the UsabilityGame, an educational game aimed at teaching usability. We conducted four experiments to evaluate the potential of learning through play, using the UsabilityGame, as well as the attractiveness of using this game as an educational resource.

UsabilityGame

The UsabilityGame is characterized as an educational game (Crawford, 1984), of the single player simulator style (Herz, 1997; Prensky, 2007), developed for the Web (available only in Portuguese at http://www.usabilitygame.com/). The game is set in a fictitious software development company, called Booble. After the company faced problems due to lack of usability of their products, Booble decided to hire usability engineers. Thus, the player (student) assumed the role of Usability Engineer and was required to perform well in each stage of the game. The stages in the game were related to Mayhew’s (1999) usability engineering life cycle. There was also the goal, albeit unspoken, of gaining the best position in the class rankings, which promoted competition among the players.

The teacher takes on the role of Usability Engineer Leader who guides students to the next steps (sometimes in an automated way through a game character) while evaluating and monitoring the students’ performance (in a specific environment). Furthermore, the teacher configures some parameters of the game, such as the game stages that are accessible to the player. Figure 2 illustrates the game flow with the steps to be followed. The game starts with a video in which the president of the Booble Corporation presents the company and explains his main problem: the large number of complaints about the difficulty of using the systems developed by the company (thereby creating the context of the game). Finally, the company’s president tells the players that he intends to open a new department in the company that is responsible for improving usability of the products, and is hiring staff for this purpose.

There are three stages of the game. Stage 1 and 2 involves two different Booble Corporation clients and their information systems. All three stages are designed to educate the players (students) about the Usability Engineering Lifecycle (Mayhew, 1999): analyzing usability requirements, developing prototypes using usability requirements, and defining usability heuristics to evaluate a prototype or product.

• Stage 1: The “Bank” client needs help with their information systems. Players analyze all scenarios, and from this analysis, define the requirements.
• Stage 2: The “Cinema” client needs help developing a system so that their customers can easily purchase tickets online. Players use a requirements document to design and test either a low-, medium-, or high-fidelity software prototype.
• Stage 3: This phase uses a variety of real-world websites so students can evaluate usability heuristics. (This stage uses Nielsen’s usability heuristics [1994].)

Figure 2. UsabilityGame flow.

The player (student) registers to use the game by completing a new hire form for the fictitious Booble Corporation. From this registration, the teacher (as usability engineer leader) links their students’ entries to a team, sets up the stages of the game, and forms a group that will compete in the class ranking.

Support Environment

In the game there are three support environments that help guide players through the three stages of the game. Figure 3 illustrates the following support environments:

• “Room of the consultant Nielsen”: This is where players access information about the usability heuristics (Figure 3A).
• “Booble Corporate President’s room”: This is where the president of Booble reviews player’s performance for each stage of the game. This character also provides feedback to help players improve (Figure 3B).
• “Room of the Usability Engineer Leader”: If a player feels “lost” in the game, he or she can go to this environment to receive tips for the next steps (Figure 3C).

Figure 3. Support environments.

Stage 1: Analyzing the Usability Requirements

The first stage of the game is associated with Mayhew’s Requirements Analysis. In this stage, the player is presented with several images, representing scenes and situations related to a bank that is a client of the Booble Corporation. After viewing each scenario, the player selects, from several requirements, the correct usability requirements (see Figure 4). The following are the scenarios presented in sequence during the game:

• Interview transcripts with the bank teller (see Figure 4), the chief information officer, and a branch manager: The interview transcripts detail questions such as (a) what is the biggest problem encountered in the current system? (b) what do you expect from the new system? and (c) what are the characteristics of the machines currently available for the bank teller?
• Report detailing situations that occur at the bank, such as customers making deposits and the existence of customer queues
• Current bank system interface: the initial system interface and deposit interface
• Storyboards representing procedures: bank deposit, rebate check, and change password
• Profile report of bank tellers: name, age, hiring date, professional education, computer knowledge
• Report summarizing the profiles of bank customers
• Document that presents the main customer complaints
• Document containing equipment descriptions that are available at the bank

Figure 4. Stage 1: Requirements analysis.

At this stage, we expect the student (player) to perceive the different techniques that can be used to identify usability requirements from the presented scenarios (interviews, environmental analysis and documents, questionnaires, among others). In addition, this stage contributes to the students’ learning by presenting examples of the requirements that are categorized according to the tasks outlined in Figure 4: user profile, contextual task analysis, consideration of technical- and platform-related capabilities and constraints, general design principles, and usability goals. This type of guidance favors the identification of components that would be defined in a style guide, as defined by Mayhew’s flowchart (1999; Figure 1).

Stage 2: Designing, Testing, and Developing a Prototype

The second stage in the game is associated with the design, testing, and development stages (see Figure 1). In this stage, players receive a usability requirements document that defines the expectations and requirements for developing a prototype. This document presents (a) user profile, (b) contextual task analysis, (c) , consideration of technical platform-related capabilities and constraints, (d) general design principles, and (e) usability goals. The following is an excerpt from the requirements document:

The Prisma 3D cinemas network wants to develop an application for Apple IPhone for the general public. Users need to have the show times of 8 branches in their own cells, which 3 of them have the 3D technology available. Furthermore, the application must allow the user to: (i) See schedule of the rooms; (ii) verify information on room availability and prices; (iii) carry out the registration as cinema customer; and (iv) purchase tickets.

We developed a tool for prototyping (Figure 5) so each player could create a low-, medium-, or high-fidelity prototype to satisfy the usability requirements of the style guide. When players design prototypes, they use and interpret their knowledge of the usability requirements to design a prototype of the software to solve a problem for a Booble client. In order to create the low-fidelity prototype, the player may include graphical elements such as squares, text, bars, and links (Figure 5A and 5C) in the editing area (Figure 5B).The teacher selects which level of fidelity prototype the player (student) must configure. When a player completes the prototype, the player requests an evaluation of the prototype by the usability engineer leader (teacher). After the teacher evaluates the prototype, the player’s score updates.

Figure 5. Stage 2: Software prototype design.

Stage 3: Evaluating Usability Heuristics

In Stage 3 of the game a variety of real-world system interfaces that are available on the Internet were presented to the players. Even though these interfaces are of real-world Internet sites, the player is told that the Booble Corporation developed them and that each site has a usability issue. By providing a variety of interfaces, students can gain a more expanded view of how usability affects many different industries. This stage correlates to Mayhew’s Installation phase (see Figure 1) where players evaluate a prototype, or as in this case, a web or software interface. After the player views the website, the player chooses an appropriate usability heuristic from a list. In that list, we used the following set of ten usability heuristics developed by Nielsen (1994):

• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation

This stage provides a method for players to exercise a heuristic evaluation by seeing a common and accepted heuristic and applying that heuristic to an interface or a real-world situation. At the end of this stage the game updates the players’ ranking. The winner was the player who accumulated the most points in all three stages. The following are some examples in the game of websites that players viewed to evaluate and address usability issues by correlating the appropriate usability heuristics developed by Nielsen:

• Real estate website: “aesthetic and minimalist design” and “match between system and the real world”
• Weather website: “visibility of system status” (see Figure 6)
• Government website: “flexibility and efficiency of use” and “error prevention”
• Academic system: “recognition rather than recall” and “help users recognize, diagnose, and recover from errors”
• E-commerce website: “match between system and the real world,” “consistency and standards,” and “help and documentation”

Figure 6. Stage 3: Evaluation example.

The Role of the Teacher

Finally, it is important to highlight the role of the teacher—or the Usability Engineer Leader, as she or he is referred to in the game. The teacher must (a) define the class that will participate in the game, thus enabling student access, (b) set up the game according to the student’s learning objectives (a teacher activates stages that he or she wants each student to play and sets the level of fidelity prototype), (c) conduct the evaluation of the prototypes (Stage 2), and (d) monitor the performance of the class.

UsabilityGame Evaluation

To evaluate the game, students performed experiments using the UsabilityGame in some classes. We attempted to identify whether using the game in a classroom setting was effective to achieve the goal of assisting in learning of how to recognize usability requirements, produce prototypes that encourage good usability practices, and recognize and apply usability heuristics to websites or other programs. Through these evaluations, we attempted to measure the effective contribution of the game to the field of HCI. In this context, we established the hypotheses in Table 1. 

Table 1. Hypotheses

Null hypotheses[1]	Alternative hypotheses
H01: Students who use the proposed game do not show any improvement in learning the concepts of usability, compared with students who do not use the game.	HA1: Students who use the proposed game show improvement in learning the concepts of usability, compared with students who do not use the game.
H02: Students who use the proposed game do not obtain a better understanding (comprehension level of Bloom’s taxonomy[2]) of usability requirements analysis than students who do not use the game.	HA2: Students who use the proposed game obtain a better understanding (comprehension level of Bloom’s taxonomy) of usability requirements analysis than students who do not use the game.
H03: Students who use the proposed game do not apply (application level of Bloom’s taxonomy) prototyping concepts better than students who do not use the game.	HA3: Students who use the proposed game apply (application level of Bloom’s taxonomy) prototyping concepts better than students who do not use the game.
H04: Students who use the proposed game do not apply (application level of Bloom’s taxonomy) heuristic evaluation concepts better than students who do not use the game.	HA4: Students who use the proposed game apply (application level of Bloom’s taxonomy) heuristic evaluation concepts better than students who do not use the game.
H05: Students who use the proposed game do not obtain a better understanding (comprehension level of Bloom’s taxonomy) of the usability life cycle than students who do not use the game.	HA5: Students who use the proposed game obtain a better understanding (comprehension level of Bloom’s taxonomy) of the usability life cycle than students who do not use the game.
H06: The use of the proposed game does not make the learning process of usability more attractive.	HA6: The use of the proposed game makes the learning process of usability more attractive.

 

Design of the Experiments

For the hypotheses analysis, four experiments were performed in a classroom. All the experiments involved a pretest and post-test. The post-test questions were the same as in the pretest, but we changed the order of the questions and alternatives. The test involved 16 questions: four questions about the usability life cycle (comprehension level of Bloom’s taxonomy), four questions regarding the requirements analysis (at the level of comprehension of Bloom’s taxonomy), four questions on prototyping (application level), and four questions on heuristic evaluation (application level).

Experiment 1

We conducted the first experiment for the UsabilityGame evaluation using students in an Information System course; these students were in their eighth semester in the discipline of Interface Design. Seven students participated. All students took the pretest, played the game, and then completed the post-test. Only one participant showed no improvement in performance after using the UsabilityGame. The other participants improved their performance by at least two points. The experiment results are shown in Figure 7.

Figure 7. Comparative performance of students in Experiment 1.

Experiment 2

We conducted the second experiment of the UsabilityGame evaluation using students in an Information Systems course; these students were in their fifth semester in the discipline of Software Engineering. Eleven students participated. Each student took a pretest and a post-test, but we separated the students into two groups. We assigned six students to the experimental group who played the game, and we assigned five students to the control group who played a placebo game of the Monopoly board game. All participants in the experimental group improved their performance after playing the UsabilityGame. In the control group, three participants did not improve their performance, and two had the worst performance in the post-test. The results are shown in Figure 8.

Figure 8. Comparative performance of students in Experiment 2.

Experiment 3

We conducted the third experiment of the UsabilityGame evaluation using students in an Information Systems course; the students were in their eighth semester in the discipline of Software Quality. Eleven students participated. Each participant took a pretest and post-test, but we separated the students into two groups. We assigned six students to the experimental group who played the game and five students to a control group who conducted a case study with activities similar to the game. In this experiment, two participants in the control group and three in the experimental group did not improve their performance after the intervention. The results can be seen in Figure 9.

Figure 9. Comparative performance of students in Experiment 3[3].

Experiment 4

We conducted the fourth experiment of the UsabilityGame evaluation using students in a Computer Science course; the students were in their eighth semester in the discipline of Usability Engineering. Eighteen students participated. Each participant took the pretest and post-test. We assigned ten students to the experimental group who played the game, and we assigned eight students to the control group who conducted a case study with activities similar to the game (same procedure as in Experiment 3). In this experiment six participants in the control group and six participants in the experimental group did not improve their performance after the intervention. The results are shown in Figure 10.

Figure 10. Comparative performance of students in Experiment 4.

Results of the Experiments

This section presents the results of the four experiments with the game UsabilityGame. The Learning Outcomes section presents the results for quantitative analysis using statistical analysis of the results from the pretest and post-test. The Student Perceptions of the Educational Effectiveness of UsabilityGame section presents a qualitative analysis.

Learning Outcomes

For statistical analysis of the results of the experiments we used the Wilcoxon test, a nonparametric test that is useful for comparing two populations (as in the case of the experiments related to the UsabilityGame).

Table 2 shows the comparison of the results of the pretest and post-test, for the 29 students in the experimental group (who played the UsabilityGame in four experiments) and for all the students who did not play the UsabilityGame (i.e., they played a placebo game or participated in an activity in the classroom addressing the same topic as that covered in the game).

Table 2. Quantitative Analysis by Topic

		Total	UL	RA	P	HE
Experimental group	Z	-3.006a	-.197a	-2.223a	-.755a	-2.865a
	p-value	.003	.844	.026	.450	.004
Control group	Z	-.201b	-.528b	-.166a	-.586a	-1.265b
	p-value	.841	.597	.868	.558	.206

Note. UL = Usability Lifecycle, RA = Requirements Analysis, P = Prototyping, and HE = Heuristic Evaluation
^aBased on negative ranks
^bBased on positive ranks

 

We see in the overall evaluation (Total column in Table 2) that the value of sig or p-value is 0.003, which is less than 0.05; therefore, we reject the null hypothesis H01 and accept the alternative hypothesis HA1, which states that “students who use the proposed game show improvement in learning the concepts of usability, compared with students who do not use the game.”

However, when we analyze the results in terms of the content addressed by the game, we see that the students in the experimental group showed significant improvement in contents related to the requirements analysis and heuristic evaluation. Given these results, we rejected null hypothesis H02 and accepted alternative hypothesis HA2, which states that “students who used the proposed game obtained a better understanding of usability requirements analysis than students who did not use the game.” We also reject the null hypothesis H04 and accept alternative hypothesis HA4, which states that “students who used the proposed game, applied heuristic evaluation concepts better than students who did not use the game.” The results also show that it was not possible to reject the null hypotheses H05, which states that “students who used the proposed game did not obtain a better understanding of the usability life cycle than students did not use the game.” Moreover, we could not reject hypothesis H03, which states that “students who used the proposed game did not apply prototyping concepts better than students who did not use the game.”

Finally, we analyze the results of each experiment for each topic addressed in the game, as shown in Table 3. Based on the results, we see that in Experiment 1, the overall result showed that the game provided an improvement in students’ knowledge, highlighting the topic related to the requirements analysis. In Experiment 2, the experimental group showed an improvement in the overall outcome, particularly in the area of heuristic evaluation. Meanwhile, the control group that carried out the placebo activity did not show any positive results. In Experiments 3 and 4, we did not obtain values that could demonstrate an improvement in learning in any of the topics, either in the experimental group or in the control group, which performed a usability case study. 

Table 3. Quantitative Analysis by Experiment

			RA	P	HE	UL	Total
Experiment 1		Z	-2.428a	-.577a	-1.667a	.000b	-2.209a
		p-value	.015	.564	.096	1.000	.027
Experiment 2	Experimental group	Z	.000b	-1.890a	-2.060a	.000b	-2.214a
		p-value	1.000	.059	.039	1.000	.027
	Control group	Z	.000b	.577a	.000b	-.447c	.000b
		p-value	1.000	.564	1.000	.655	1.000
Experiment 3	Experimental group	Z	-1.000a	-1.633c	-1.069a	.000b	-.412a
		p-value	.317	.102	.285	1.000	.680
	Control group	Z	.000b	.000b	-.378c	-.276a	-.406a
		p-value	1.000	1.000	.705	.783	.684
Experiment 4	Experimental group	Z	-.000b	-.707a	-.647a	-.264a	-.704a
		p-value	1.000	.480	.518	.792	.481
	Control group	Z	-.264a	-.535a	-1.732c	-.707c	-.632c
		p-value	.792	.593	.083	.480	.527

Note. RA =Requirements Analysis, P = Prototyping, and HE = Heuristic Evaluation, UL = Usability Life Cycle.
^aBased on negative ranks
^bThe sum of negative ranks equals the sum of positive ranks
^cBased on positive ranks

Student Perceptions of the Educational Effectiveness of UsabilityGame

The 29 participants who played the UsabilityGame answered a questionnaire with 10 questions concerning their perceptions of the game:

• Was the game attractive and interesting to play?
• Did the game teach analysis of usability requirements?
• Did the game teach prototyping?
• Did the game teach heuristic evaluation?
• Did the game teach usability engineering life cycle?
• Did the game provide enough information about usability?
• Did the game introduce a degree of difficulty appropriate for learning?
• Was the length of the game appropriate?
• Was the idea of teaching usability through a game simulator adequate?
• Was the contextualization presented by the game adequate?

The responses observed the Likert scale, and the participant had to answer 1 if they strongly disagreed, 2 if they partially disagreed, 3 if they neither agreed nor disagreed, 4 if they partially agreed, and 5 if they strongly agreed. The results of the questionnaire (Figure 11) served to assess, among other respects, hypotheses H06 and HA6 in relation to the attractiveness of the UsabilityGame.

Analyzing the results of Q1 (which questioned the player on the attractiveness and whether the proposed game was interesting to play), we found that around 75% of the students responded that they strongly agreed or partially agreed. Based on this result, we can accept hypothesis HA6 and reject null hypothesis H06, which states that the proposed game makes the process of learning the theme of usability more attractive.

Figure 11. Qualitative analysis.

It is worth highlighting the questions that presented greater divergence (Q3 and Q8). We believe that these results are related because the students were given approximately 1 hr 30 min to finish the game, but the prototyping stage consumed most of the time. Moreover, the resources available in the UsabilityGame for prototyping were less than those found in traditional prototyping tools, a fact that may have led to this result.

Conclusions

Because the systematic mapping did not enable us to find any serious games related to the area of HCI, this paper presented a simulation game to support the teaching of usability to students of undergraduate computer courses. The game environment was designed to practice concepts related to usability. The UsabilityGame addresses the usability life cycle. It is divided into three stages, the first covering usability requirements analysis, the second focusing on prototyping, and the third addressing heuristic evaluation.

Four experiments were conducted to evaluate the game. The UsabilityGame obtained positive results when we analyzed the overall result of all the students who played the game. In this context, we can affirm that the game helps students learn some concepts about usability, with greater emphasis on issues of requirements analysis and heuristic evaluation. Moreover, the game had good results in relation to attractiveness in the learning process of usability (as noted in Table 4).

On the subject of prototyping, we did not find evidence that the game helped learning. This may be due to the restrictions of the tool for prototyping in the game, as the students reported that they were bored with the limitations of the tool. This conclusion is reinforced by the qualitative results concerning the prototyping phase of the game, because when asked about their learning of the area of prototyping, nearly 40% of the students said they disagreed totally or partially, a result that is contrary to those of the other areas addressed in the game.

We also cannot conclude that the UsabilityGame effectively taught the sequence of the usability engineering life cycle. We assumed that this result is associated with the fact that the game does not directly address the issue, but relates it only to the sequence of phases of the game (which observed the life cycle levels proposed by Mayhew, 1999). Another possible reason for this result is the fact that the game does not “force” the player (student) to play the phases in sequence, that is, it allows the player to freely choose which phase they wish to play each time.

Table 4. General Analysis of the Hypotheses

Hypothesis	Exp. 1	Exp. 2	Exp. 3	Exp. 4	General
H1: Students who use the proposed game show improvement in learning the concepts of usability, compared with students who do not use the game.	yes	yes	no	no	yes
H2: Students who use the proposed game obtain a better understanding (comprehension level of Bloom’s taxonomy) of usability requirements analysis than students who do not use the game.	yes	no	no	no	yes
H3: Students who use the proposed game apply (application level of Bloom’s taxonomy) prototyping concepts better than students who do not use the game.	no	no	no	no	no
H4: Students who use the proposed game apply (application level of Bloom’s taxonomy) heuristic evaluation concepts better than students who do not use the game.	no	yes	no	no	yes
H5: Students who use the proposed game obtain a better understanding (comprehension level of Bloom’s taxonomy) of the usability life cycle than students who do not use the game.	no	no	no	no	no
H6: The use of the proposed game makes the process of learning usability more attractive.	yes	yes	yes	yes	yes

We highlight in Table 4 a lack of positive results in Experiments 3 and 4. In both cases, the control groups performed activities related to subject usability (contrary to Experiments 1 and 2, in which there was no control group, or a placebo was applied). Thus, we cannot affirm that the game is better than another traditional activity, although we can state that it may be just as efficient. However, we must consider the results shown for H6 that highlights that the UsabilityGame is more attractive as a tool in the process of teaching and learning.

Finally, the results obtained so far motivate the development of improvements in the game, such as functionality for teacher change scenarios and case studies available in the game, and the re-structuring of the prototyping stage. We also plan to conduct more experiments with different design and evaluation artefacts.

Tips for Usability Practitioners

The following findings, based on the results of this study, have practical value for usability practitioners:

• The systematic mapping indicated a need for empirical research related to teaching HCI. So, in teaching concepts related to usability, propose new methods and empirically evaluate the results.
• A point to be highlighted is that despite the extensive use of games for teaching computing (shown at the beginning of the Teaching Usability and Serious Games section), we did not find any studies related to the development and use of games for teaching usability. Use of a game makes the learning process of usability concepts attractive.
• UsabilityGame is a gaming option to learn, train, or test the skills in the field of usability.
• In accordance to the observed results, UsabilityGame can be improved. Especially when it comes to prototyping and understanding the usability life cycle.

[1] Trochim and Donnelly (2006) proposed that the hypothesis that supports the prediction is the alternative hypothesis, and the hypothesis that describes the remaining possible outcomes is the null hypothesis. They recommended using a notation like HA to represent the alternative hypothesis and H0 to represent the null case.

[2] Bloom’s taxonomy is a hierarchical organization structure of educational goals (Bloom, 1956). This includes the recall or recognition of specific facts, procedural patterns, and concepts that serve in the development of intellectual abilities and skills. There are six major categories of cognitive processes, starting from the simplest to the most complex: Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation.

[3] Students in the control group received a case study similar to Stage 2 of the UsabilityGame from which they conducted an analysis of usability requirements and developed a prototype on paper based on the requirements. The students then conducted a usability test of that prototype.

Published in: in Volume 11, Issue 1,