Rhetorical Grounding and an Agile Attitude: Complex Systems, Multi-Genre Testing, and Service Learning in UX

As PhD students in a hybrid online/residence program, we were assigned to be a usability testing team in a service-learning project where we experienced the excitement and challenges of rapid-prototyping and small-shop product development. The product inventor/manager and designers were members of our academic unit, and one of the investors was our instructor. Some of our student and faculty colleagues were considered potential real-life customers. Still and Albers (2010) assert that, “caught so often in the middle between products and their consumers, regardless of the environment, we find it almost an essential part of our job to always be ready to embrace and operationalize new methods or theories” (p. 190). Indeed, as student usability testers, we were “caught” among the product developers and investors, potential clients, graduate faculty, and a condensed timeline. We were also caught in a methodological bind because our contexts did not allow for the smooth application of theories and practices as explained in the UX scholarly literature. Our usability testing experience demanded quick, sometimes improvisational work even as we struggled to establish our footing on usability studies ground. As an outcome of that experience, this article reports and reflects on two issues: (a) how usability researchers both respect and subvert a discipline’s “best practices” based on the rhetorical nature of usability testing and (b) how embedded service learning can be a strong academic and industry bridge connecting the ideals of theory and best practices to the practical realities of serving a client.

The six of us met as we attended our PhD program’s annual residential requirement, a compressed two-week semester during which we completed a three-credit course in usability studies. Although one of us worked with clients on website development, none had previous coursework in usability testing. We met at least four hours per day, six days per week in addition to other in-residence requirements such as attending scholarly presentations and professional development sessions. Time constraints meant the usability formal presentation and report were due to the client at the last meeting of the abbreviated timeline.

In the service learning component of our course, a start-up company, owned by a faculty member in our graduate program and staffed by two other graduate students (one in our program and one in computer science) asked our team to perform a usability test on its new product—an eye-tracking product designed for usability testing practitioners. The product is a tri-part, complex system: a camera and light device worn on the head, a capture program to record eye movements of participants, and analysis software that reveals eye-movement patterns in the screen captures. Conceived as a lower-cost competitor in a market dominated by high-end ($20–30K) systems, the product was on a fast-track schedule to be unveiled at a conference and to go into retail production in less than a month, and our team was expected to complete usability tests and give recommendations that the clients could act on immediately. In fact, as we were testing the prototype provided on the first day of the project, a second-version prototype was concurrently being developed. The designers often responded to our informal, in- process testing results overnight.

The complex nature of the product called for a complicated set of tasks, including concurrent hardware and software testing (headgear and user interface), an approach we call “multi-genre” testing. From the initial meeting with the client until our final recommendations, the situation demanded that we extend theoretical spaces in usability testing in order to balance the needs of the client with unique circumstances and a short time frame. Our test design—including tasks, scenario, and script—required us to understand and acknowledge tensions in the UX field. As each contextual challenge arose, we sought practical but justifiable solutions pushing theoretical boundaries.

In this article, we explore the complex nature of our project on theoretical and practical grounds, focusing in part on the rhetorical features of the situation. We then describe the effects such complexity had on the design of our usability testing procedure, the outcomes of the test, and the lessons we learned regarding agility and the value of a service project bridging the academic and industry divide.

As Redish (2010) noted: “Understanding complexity—how to create products and processes for complex situations as well as how to build usability in and assess the usability of complex systems—is becoming a major topic for technical communicators and usability professionals” (p. 198). Although we did not realize it at the time, because we were focused on meeting both graduate instructor and client demands, our academic grounding in rhetoric and technical communication served as an excellent nexus to cope with a sprinter’s deadline, a complex system, and a constantly shifting path. The rhetorical situation, defined by Bitzer (1968) as a “natural context of persons, events, objects, relations, and an exigency which strongly invites utterance” (p. 5), provided the assumptions and constraints under which our team worked.

The “natural context” included the convergence of client, designers, and testers during the two- week seminar. The service-learning project produced our exigency, and the client’s authentic need for user experience feedback provided the kairotic moment. Based on an expected level of user experience, the graphical user interface (GUI) was built with no help files or user’s manual. Designers felt the GUI was intuitive, expecting the product’s target audience—usability practitioners—to have pre-established mental models based on a popular usability-related software package already in use. Ideally, to test whether or not users had semantic structures to intuitively use the GUI, we would have given participants a Bitzer-like exigency in the form of a realistic task assignment. In other words, if our target users were UX testing professionals, then they could have already known how to competently use the prototype under development. Instead, because the participants were to interact with the prototype for the first time, we created a scenario, script, and tasks that were more exploratory and open-ended, allowing participants to create their own meaning—a rhetorical situation more aligned with Vatz’s (1973) idea that rhetorical situations are entirely subjective, that each rhetor creates the exigency (rather than wait for a natural occurrence) and then crafts a response to it. Adopting this approach to usability testing, we found the spaces between the objectivity of Bitzer and the subjectivity of Vatz. Our foundational positioning supports Yates and Orlikowski’s (2002) view “that researchers should turn their attention to the active shaping of kairotic moments” (Artemeva, 2005, p. 394) rather than passively accept the natural situation predicted by the client.

Another element of rhetorical theory is genre. When an “utterance” or response is made, it must come in some form or another. More formally, genre is defined as “typified symbolic actions in response to recognizable situation types… [that] constitute human activity” (Artemeva, 2005, p. 392). “[G]enres do not exist independently… [but rather] must be understood in relation to other genres and their interplay” (Artemeva, 2005, p. 393; see also Devitt, 2000). The way that an actor responds to a rhetorical situation is shaped by the forms of communication already in play. Complicating the rhetorical situation for our testing, the product contained three components, each one in a different stage of development (see Table 1). The hardware was a late stage prototype, and the GUI was in what Still and Morris (2010) described as a medium-level phase. The GUI had a basic look-and-feel already established and was coded for basic functionality, but it included no user support (no quick-start guide, no wizard, no built- in help). According to Lepore’s (2010) design continuum, the system was inclined towards the “prototype” level but still in the refinement or “dress rehearsal” phase. It included basic structure and flow but lacked details that would allow formal prototype testing. User documentation (a quick-start guide, help menu, embedded FAQs, or wizards) was not present. Inherent tensions were created through attempting to test the hardware, software, and documentation simultaneously. Despite being at different stages of the development process, the system could not be divided and independently evaluated as stand-alone components.

As a further complication, the product was designed to have two simultaneous audiences or users: a usability lab professional and a testing participant. The usability lab professional would fit the headgear/camera over the participant’s eye, calibrate the hardware to the software using the GUI, and start the eye-tracking process. However, because the product was designed to fit the functionality and pricing structure needs of a “small shop” usability lab, we could not presume a two-user situation. As Redish (2010) argued, “Complexity is not so much an attribute of a product or process itself as it is an attribute of the interaction between that product or process and its users. Thus, complexity is audience specific” (p. 199). Although the product system did not necessarily meet all of Redish’s (2007) list of complex system characteristics, the hardware and GUI components at different stages of development and the compressed usability testing period created overall system complexity. We likened these different stages of development (and the correlation of development stages to different standards of usability testing) to a challenge of having multiple genres but only one rhetorical situation.

Thus, when presented with the rhetorical situation, including the testing variables and the client’s needs, we could not rely on a single genre of usability testing. UX scholarship has generally either focused on the genre of early prototype testing or the genre of late prototype testing. The situation (exigency) driving our project was the faculty/client’s need for immediate feedback, and we were not in a position to ask the designers to stop their work on the hardware and software in order to produce documentation. As students, we did not have the agency to align the levels of development of the various components. Therefore, we turned to our shared background in rhetoric and critical thinking in order to adapt the testing design to the complex product.

One primary effect of our testing’s rhetorical situation was a struggle with the design of our think-aloud protocol. As Nielsen (1993) noted, “Thinking aloud may be the single most valuable usability engineering method” (p. 195), and concurrent think-aloud protocol (CTA) methods have generated significant discussion regarding rigor and best practices, particularly regarding tester-participant interaction. Ericsson and Simon (1984) originally argued that rigor is primarily achieved when the tester fades into the background and resists intrusion. Nielsen (1993) was pragmatic, allowing tester interference in cases where the user was stuck or was encountering problematic areas consistently encountered in previous tests. Tamler (1998) suggested designing a test’s CTA based on testing goals and recommended probing questions, active listening, targeted user guidance, and user design input as potentially useful in qualitative studies. An alternative approach grew out of speech communication theory and the notion that “the issue in usability testing becomes not one of disappearing from participants, but rather one of creating a highly asymmetrical speaker/listener relationship” (Boren & Ramey, 2000, p. 267), placing the product in the “subject” position and the participant in the “expert and primary speaker” position. Makri, Blandford, and Cox (2011) reflected on a variety of think-aloud approaches, highlighting decisions concerning when the thinking occurs, either during or after the tasks. Although emphasizing the need for authenticity in their testing’s rhetorical situation by making the test sessions “as true to life as possible (within our study’s constraints)” (p.341), Makri, Blandford and Cox did prompt for data, thereby aligning with Boren and Ramey (2000) as well as with Tamler (1998) on the debate concerning researcher interference.

The product designers provided the team with two key factors affecting product design, both of which influenced the product’s rhetorical context and, therefore, the study’s design: (a) the menus were structured to match those of TechSmith’s Morae in order to enhance ease of use for usability experts familiar with that software and mental model, and (b) the product’s price point was intended to increase the accessibility to eye-tracking systems for new or small usability labs, as well as individuals interested in eye-tracking for other fields. Considering these two assumptions together revealed a likely problem: new, small-shop, or extra-disciplinary users may not have a familiarity with the Morae mental model. Nielsen (2010) described such a gap in designers’ and users’ mental models, where the system may have a wonderfully rich model that is inefficient for users who each apply a different mental model when navigating that system and predicting what selections to make in the completion of a task. This was one of our concerns from early in the process and influenced our participant screening.

Scholars have debated the appropriate number of participants in usability tests (Caulton, 2001; Faulkner, 2003; Nielsen, 2000; Spool & Schroeder, 2001). Caulton (2001) described two types of variables affecting the number of users tested: user group and problem type. The product users have a common, clearly-defined background with a certain level of computer literacy, so our research sample could be homogeneous. In addition, because we sought a broad-based understanding of the product, we were looking for shared (not unique) problems. For an accurate representation of the target user group, we recruited six testing participants from our graduate program: two faculty members, two doctoral students, and two master’s degree students. Academic research institutions are one of the product’s target markets, helping to support the validity of finding participants in this pool. We also ensured that all participants had no prior experience with any iterations of either the hardware or software being tested. Participants volunteered their time and were not compensated. The department’s usability lab was covered under a blanket Institutional Review Board/Human Subjects Review approval.

Although homogenous in broad disciplinary interests, they had varying levels of usability experience. Participants averaged a 3.67 on a 5-point scale in self-reported usability software expertise (where 5 is extensive), with at least two participants having had direct experience with the Morae software. In contrast, participants reported very little experience with eye tracking, an average of 1.5 on the same 5-point scale. Without experience in eye tracking, the expectation was that users would apply their pre-existing familiarity with other usability technologies (i.e., mental models) in order to navigate the product. Formal mental model matching approach can be too time-consuming for usability study (see Sutcliffe, Ryan, Doubleday, & Springett, 2000), so we employed a more concise method where participants were asked to freely explore the GUI for 3 minutes as they talked aloud, performed a series of tasks, then reported menu expectations and recalled navigation in a retrospective questionnaire (see Table 2).

Although we wanted our testing to be as natural to the rhetorical situation as possible, the product’s levels of development were limited to the scenarios or tasks we could test. Our scenario took advantage of the participants’ familiarity with usability testing but less common knowledge of eye-tracking systems. Scripted and delivered by the same facilitator to all six participants, it began with this introduction:

Because these tasks use directive wording, we acknowledge that we were partially testing the clarity of the task instructions. Without any drafted documentation (and because of the potential for mental-model mismatch), telling participants to complete high-level tasks without providing sub-steps was not feasible. The complexity of the system meant that providing limited direction to participants allowed the facilitator to act as a verbal “quick start guide” and focused our observations on how easily the participants were able to complete the tasks.

While these interventions conflicted with standard best practices (Barnum, 2011; Boren & Ramey, 2000), the subsequent observations seemed to generate potentially helpful feedback to the GUI and hardware designers as well as to the future documentation writers (see the Results section). Interruptions were kept to a minimum and were carefully framed to reduce bias and maintain asymmetry. For example, during task one, the facilitator asked, “Do you find commands where you expect them?” and “Do you find the menu arrangement efficient and effective for your expectations?” During task three, the participant was prompted to “tell me how the headgear feels on your head.” Other than these planned, specific questions and the token verbal cues (e.g., “mm hm”), participants were occasionally reminded that they were the experts and the product was what was being tested (as recommended in Boren & Ramey, 2000, and in Barnum, 2011).

To combat bias and to address potential questions about the reliability of our data, our team addressed a number of Creswell’s (1998) procedures and all of Hughes’ (1999) areas. First, to ensure that our test reflected actual user activities, we made the scenario as natural to the anticipated user environment as possible by taking into account the product, audience, and what that audience would do when first interacting with the product. We let the participants follow the tasks with little interruption (as per the scenario) but then sought out rich feedback when a problem was encountered. We used Morae to capture video, audio, and screen movements during the tests, and three team members encoded and cross-checked their observations. The in-room observer provided an additional perspective on body language and interaction, and the combination of recordings and observations allowed for prolonged engagement and persistent observation, triangulation, and rich description. Next, because the product tested a usability lab product, testing it in a university’s usability lab provided an inherently natural environment, which suggests transferability (Hughes, 1999). Finally, with regard to dependability, Hughes recommended safeguarding against biases by having a team formed from different functional perspectives. Although at the time, our team was all part of the same PhD program, we came from various backgrounds: three of us worked in post-secondary instruction and administration, one in both secondary and post-secondary instruction, one in state government as a technical writer, and one in digital strategy and technology development for the civic sector. The mix of expertise, as well as triangulation among our data sources, enriched our analysis as we sought to further enhance the study’s rigor.

Participant experience with hardware was mixed. Five out of six participants thought the headset was comfortable and that the camera affixed to the headset was not obtrusive. However, concerns were expressed regarding the headset’s stability (5/6) and insecurity with positioning of the hardware (4/6). An unexpected question revealed by participants was whether or not the product would work with eye problems, including corrective lenses (3/6) and cataracts (1/6). While participants understood what it meant to align the camera to their eye, the process itself did not work as designed. Participants cited problems adjusting the headset (4/6) and physical fatigue (2/6). Ultimately five out of six participants were unable to accomplish the alignment task in the allocated five minutes and required the help of a technician in order to continue exploring the product system.

Menu architecture was judged by the users as familiar and, at first glance, intuitive. Based on their think-aloud commentary and later feedback, the initial comfort resulted from the menus being based on a common standard Microsoft Windows architecture. However, the actual GUI use and attempted completion of testing tasks created uncertainty about menu order and hierarchy. Participants noted confusion about the sequence of steps. One noted “I would think tasks would be subordinate to file, the container for the project, and hence tasks would be under file instead of separate.” Although our participants had experience with usability testing, and some had direct experience with the Morae system model, they expressed unfamiliarity with the Morae-based terms: “projects,” “tasks,” and “tests.” The presence of a dark and pixelated capture screen on the computer further aggravated the situation. In short, a combination of reliance on the Microsoft Windows mental model and targeted users’ lack of expected familiarity with Morae terminology contributed to a high degree of task failure and user frustration. A lack of documentation or wizard guidance was cited as a key issue.

Perhaps the most frustrating aspect to participants was the calibration process required to establish a connection between the hardware and the software. While they reported clearly understanding the process, participants were confused about calibration and expressed negative reactions. The software offered a “Calibration Score” which showed “0” for all participants throughout multiple attempts. Participants inferred that this response was a personal failure (“It seemed to be telling me I was doing something wrong, which I probably was”). The software’s on-screen directions did not mitigate confusion. Only one participant was able to successfully calibrate the system independently, while the others required the assistance of our designated technician.

Based on the testing of this complex, multi-component system situated in a complex rhetorical situation, we recommended the addition of user documentation, emphasizing the need to define key terms and to provide step-by-step instructions for use of the software and hardware. While the client responded that our testing of the hardware prototype did offer insights into that iteration of the design, it is difficult to say whether or not that feedback was ultimately helpful to the product’s success because another iteration of the prototype was concurrently being built as we tested the one we had on-hand.

Our service-learning course was an “agile challenge” that Barnum (as cited in Six, 2011) defined in the form of a question: “How to fit usability testing into the short timeframe of a typical sprint?” (Six, 2011, “Adapting User Research,” para. 1). According to the Manifesto for Agile Software Development (Beck et al., 2001), the following are principal values of agile programming:

Our contextual role was mixed. Because the owner and designers were constantly in close contact, the testing process was highly collaborative. However, even as we were embedded contextually in the product design process, our team also had characteristics of an external usability testing organization. We completed tasks associated with an agile testing environment: fast planning, participant recruiting, rough and tumble pilots, and quick debriefs (Six, 2011). Our “contract” for the work was verbal, and although we did provide a formal testing plan to our graduate instructor, the plan was adaptable in response to the rapidly developing nature of the product. Scrums cemented our relationships in multiple directions: among ourselves as working group members, with the product designers, with the product owner, and with our graduate professor. Especially as online students attending the in-residence program for the first time, the challenge and intensity affected “social and cultural interrelationships, instilling increased cross-disciplinary empathy and understanding [and replacing] traditional document-centered communication…with fast, lightweight communication that reflects the current project reality” (Ramey as cited in Six, 2011, “Team Dynamics,” para. 4).

If we had been in a later stage of user testing for the elements of our service-learning course— hardware, software, and both interacting together—we could have perhaps met Madrigal and McClain’s (2011) expectation for “testing a version or mockup that is free of glitches, bugs, or known errors” and emphasizing the importance of testing for a product in “primetime” (para. 1). We attempted to find an intermediary solution, particularly looking at lo-fi options that could draw insights and provide a less frustrating situation for the end user. However, given the number of factors that required testing and the end-goal of having a more usable product to introduce at a rapidly-approaching conference, these options were not feasible.