Abstract

Collaborative writing technology is utilized in the workplace to co-author documents. Unfortunately, this technology is not accessible or usable for persons who are blind. As a result, persons who are blind cannot participate in collaborative writing that is critical in business and in collegiate environments. In order to improve the accessibility and usability of collaborative writing technology, a Microsoft Word add-in prototype was designed, developed, and tested using an iterative design approach involving two rounds of one usability study. Eleven participants, who are blind with no residual vision, participated in the usability studies and provided feedback and suggested improvements based on their experience while interacting with the Word add-in prototype. The prototype was modified based on the suggested improvements from the participants after each round of the usability study. Before the second round of the usability study, the dependence on the Word ribbon menu was replaced by the utilization of Windows message boxes for presenting, accepting, and rejecting document revisions and comments. The participants of both rounds of the usability study “agreed” or “strongly agreed” that the Word add-in prototype interface was clear and understandable, easy to use, improved their performance, and enabled the tasks to be completed without any problems. Also, the participants were satisfied with the time that it took to complete the study tasks, and they would utilize the Word add-in interface in the future on a regular basis.

Tips for Usability Practitioners

Developing accessible and usable collaborative writing interfaces for participants who are blind, consider the following tips:

• When presenting document revisions or comments to a user, consider providing the paragraph or sentence context to improve understanding. In this case, the paragraph context of each revision or comment was provided in order to improve understanding.
• Develop the interface so it interacts exclusively with a standard keyboard and screen reader. This will enable persons who are blind and cannot access and use a mouse to interact with the interface.
• Test the interface with a screen reader, such as JAWS, to verify that persons who are blind will understand the interface.
• Prior to evaluating collaborative writing prototypes, it is important to present an overview of how the application works and allow the participant to “walk- through” some scenarios in order to become acclimated with the application’s functionality.

Introduction

According to the World Health Organization (WHO), “285 million people are visually impaired worldwide: 39 million are blind and 246 have low vision” (2013). Collaborative writing is essential to many organizations in order to exchange ideas and compose a complete document. Unfortunately, persons who are blind have encountered accessibility and usability issues when interacting with popular collaborative writing tools such as Microsoft Word and Google Docs (Schoeberlein & Wang, 2012). For example, persons who are blind are not able to identify whether a given document is in its final version or still contains modifications. Even when they were told there are modifications, persons who are blind had a hard time identifying the context of the modifications and comments in a document.

How should a collaborative writing interface be designed so that it is accessible and usable for persons who are blind? In order to answer this research question, a Word add-in prototype was designed, implemented, and evaluated using an iterative approach that included two rounds of design and usability sessions in a usability study.

This paper presents the design and evaluation process conducted. The paper starts with a brief discussion on the related research efforts and the challenges persons who are blind are facing when using the collaborative writing features in Word. Detailed discussion on the study design is presented next, followed by the discussion of the two rounds of design and usability evaluation of the Word add-in prototype (Prototype 1 and Prototype 2) with a comparison of the two prototypes. Future research plans, conclusions, recommendations, and the tips for practitioners are presented at the end.

Related Research

Screen readers read the text that is displayed on a computer screen with a speech synthesizer. A screen reader is the interface between the computer’s operating system, its applications, and the user (Hampel, Keil-Slawik, Claassen, Plomnann & Reimann, 1999; Pitt & Edwards, 1996). It is among the most popular form of assistive technology especially for blind or visually impaired users. Other forms of assistive technology being used by blind users may include refreshable Braille display or Braille output (National Federation of the Blind, 2014). However, Braille output has not been widely used with its user population in decline because of the low Braille literacy and limited access to Braille technology (WebAIM, 2014).

Pitt and Edwards (1996) suggested that the amount of information, the order of presentation of information, the use of pauses and prosody, the pronunciation and speech quality, and the use of non-speech sounds are the main problems with screen readers. However, the majority of studies that have been conducted relates to web accessibility problems (e.g., Chen, Tremaine, Lutz, Chung, & Lacsina, 2006; Hermsdorf, Gappa, & Pieper, 1998; Lazar, Dudley-Sponaugle, & Greenidge, 2004). One of the problems many screen reader users experience is that not all applications are compatible with a screen reader. For example, the screen reader cannot interpret graphical information on the screen if it is not labeled properly. Poorly labeled forms and the use of frames can also make the program inaccessible to persons who are blind (Lazar, Beere, Greenidge, & Nagappa, 2003). According to a set of surveys conducted by WebAIM (2014), although there are many choices of screen readers in the market, JAWS (http://www.freedomscientific.com/Products/Blindness/Jaws) has been the most popular and commonly used screen reader selected by participants. Therefore, this research is focused on designing an interface that works with JAWS.

Dorigo, Harriehausen-Mühlbaue, Stengel, and Dowland (2011) surveyed respondents on their access and use of electronic documents in order to begin to understand the user’s point-of-view. The participants, who are blind or visually impaired, had key areas of concerns in regard to digital documents including the accessibility of digital documents, the need for an overview of the structure and content of the documents, and the usability of standardized assistive technologies (Dorigo et al., 2011). Providing accessible documents that promote awareness of the structure of a document when utilizing standardized assistive technologies becomes critical. Buzzi, Buzzi, Leporini, Mori, and Penichet, (2010) utilized a screen reader to examine Google Docs in order to understand how persons who are blind or visually impaired would interact with Google Docs. They found that “serialized content and mixed content and structure caused confusion leading to problems in user perception such as lack of context, lack of interface overview and difficulty understanding user interface elements” (Buzzi et al., 2010, p. 93). They suggested the use of the Web Accessibility Initiative’s Accessible Rich Internet Applications (WAI-ARIA) guidelines for improved screen reader operability for Google Docs (Mori, Buzzi, Buzzi, Leporini, & Penichet, 2011). Mori et al. later applied WAI-ARIA to the Google Docs interface as one approach to improving accessibility.

In order to expand on prior collaborative writing research, Schoeberlein and Wang conducted a baseline usability study examining the accessibility and usability of Word from March through May of 2012 (2012). All five participants of this study were blind with no residual vision. All participants used MS Word at least twice a week and relied on screen readers such as JAWS to interact with computer systems. The focus of the usability study was to identify the accessibility and usability issues related to collaborative writing features. Word, one of the most popular word processing software applications, was picked as the test bed for this usability study. The participants were asked to complete a set of tasks involving searching, accepting, and rejecting revisions, and reviewing comments in a Word document. Only 62% of the participants who are blind completed all of the baseline study tasks (Schoeberlein & Wang, 2012). The participants had trouble understanding the context of the revisions and comments, as well as accepting or rejecting each revision and comment contained in a document. For example, a sentence containing modification markups when using the track changes feature in Word may be presented as “A well regulatedwell-regulated militia” to sighted users. When blind users try to access the same content using the JAWS screen reader, it is read as “A deleted text well regulated inserted text well dash regulated militia.” Therefore, the participants had a hard time distinguishing whether the “deleted text” was part of the content of the document or the indication of a modification. The results from this baseline study revealed a real need for providing improved accessibility and usability of collaborative writing features such as revisions and comments.

Based on the results of the baseline usability study, a Word add-in prototype was proposed (Schoeberlein & Wang, 2013). The prototype utilized Windows message boxes to present the paragraph or sentence context for each revision and comment followed by the actual revision and comment to the user of the Word interface. The JAWS screen reader audibly presented the content of each message box so persons who are blind could interact properly with the interface. The message boxes provided interface support for visually-able users too. An initial test provided evidence that the Word add-in prototype was accessible, usable, and available for further testing and validation (Schoeberlein & Wang, 2013).

Study Design and Procedure

For this study, an iterative design approach involving two rounds of design and usability evaluations was adopted. After each round of design and implementation of changes to the prototype, we conducted a usability evaluation session. Each session ran for approximately 90 minutes. All participants for all activities on the computer used the JAWS screen reader. The usability session for both rounds were conducted following the procedure below:

1. Subscribe subjects. With help from the National Federation of the Blind (NFB) and the Maryland State Library for the Blind and Physically Handicapped, a subject-recruiting letter introducing the nature of the usability study and asking for participation was delivered to potential subjects through email.
2. Obtain subject consent form and background questionnaire. Prior to each usability study session, each participant, who was blind, was presented with an electronic version of the informed consent form. Each participant then, using a standard writing guide for persons who are blind, signed a printed version of the informed consent form. The researchers created two questionnaires (background and post study questionnaires) online in an accessible format on surveygizmo.com. The background questionnaire was designed to collect participant’s personal information (i.e., age range using an interval scale and gender using a nominal scale) and experience levels with Word, JAWS, and computers (i.e., primary use, frequency of use, and years of experience using an interval scale). The response questions included a list of choices (i.e., daily, weekly, and monthly) or a range of choices (i.e., less than 1 year, 1 to 5 years, 6 to 10 years, 11 to 20 years, 21 to 30 years, and over 30 years) to choose from. Additionally, there were open-ended questions to allow free-form participant responses. Each participant completed the survey questions online through the URL provided.
3. Conduct brief prototype training. The usability study sessions began with a brief introduction of the purpose of the study—to test the accessibility and usability of the Word add-in prototype (Prototype 1 or Prototype 2) while completing collaborative writing tasks. The introduction was followed by a brief walk-through and training of the prototype’s features for searching, accepting, and rejecting revisions and comments within a Word document.
4. Observe participants working on the tasks. After the brief prototype walk-through, each participant independently examined a sample document containing revisions and comments throughout the document body for testing purposes. The “Bill of Rights” (United States Government Archives, 2011) was used as the testing document due to the fact that most participants were familiar with its content. Participants completed the following four tasks associated with collaborative writing activities:
5. Browse through all of the revisions.
6. Accept a specific revision requested by the researcher.
7. Reject a specific revision requested by the researcher.
8. Accept or reject one remaining revision selected by the participant.

While the participants were performing the tasks, the researcher recorded task performance in terms of the successfulness and time of task completion for each task. A “success” mark was recorded for each task that was completed successfully. The time stamps for both beginning and ending each task were recorded using a standard stopwatch. Task completion time was determined by subtracting each task’s begin time from its end time. In addition, the researcher recorded observations, in a notebook, of the participants’ performance of each task.

5. Distribute the post-study questionnaires. After finishing all the tasks, the participants filled out a brief post-study questionnaire on their experience and opinions in regard to the prototype’s interface. Same as the background questionnaire, the post-study questionnaire was also presented online in an accessible format on surveygizmo.com. The response questions in regard to the participants’ experience with the prototype utilized a Likert-type scale with values such as Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree.
6. Collect feedback and debriefing. Upon the completion of the post-questionnaire, the participants were showed further tips on how to use the prototype and more discussion was conducted to collect their opinions. All the discussions were audio recorded.
7. Transcribe audio recordings and data analysis. Following each usability study, transcripts for the audio recordings were made and the researcher’s notes were collected. Content analysis of the audio transcript was conducted by applying coding to identify unique categories for each statement in the transcripts. In order to ensure the reliability of the coding, two coders examined and coded the transcripts independently after a brief training session on how to code a transcript. The coding from both coders was compared to check for consistency in order to validate the categories identified. Cohen’s Kappa coefficient was utilized to validate the agreement between the two coders as to the meaning of the qualitative data. Agreement of 70% or more was utilized as an indication that the interpretation of qualitative data is consistent between coders. The data collected from the post-study questionnaires and task completion records was put into IBM’s SPSS predictive analysis software to conduct the analysis.

Research Environment

Both rounds of the usability studies were conducted using a Dell Latitude E6400 laptop computer running Windows 7 Enterprise. The installed applications consisted of Word 2010, JAWS screen reader (Version 13), and the developed Word add-in prototype, either Prototype 1 or Prototype 2. The external hardware consisted of a standard keyboard.

Participants

All participants were blind with no residual vision. There were four males and seven females. The ages of the participants were distributed almost evenly from 19 years to over 60 with the biggest group of participants, 36.4%, being between 50 to 59 years of age, followed by 27.3% of them in their 40s, 18.2% between 19 to 29 years old, and 9% each in their 30s and above 60.

The participants’ working experience spanned from 6 years to over 30 years. They had at least 6 to 10 years of computer experience, at least 1 to 5 years of experience with Word, and at least 6 to 10 years of experience with the JAWS screen reader. All of the participants utilized the computer daily, and the majority of the participants (7, 63.6%) utilized the computer at work. The majority of the participants (5, 45.5%) had 11 to 20 years of experience and four participants (36.4%) had 21 to 30 years of experience utilizing the JAWS screen reader. All of the participants had at least one year of experience utilizing Word. The majority of the participants (8, 72.7%) utilized Word daily. The broad age ranges and experience levels of the participants of the usability study provide a good mix of age and computer and screen reader experience.

All 11 participants volunteered to participate in the two rounds of the usability study. Five participants participated in the first round with the original Word add-in prototype (Prototype 1), and the other six participants participated in the second round with the modified Word add-in prototype (Prototype 2). Each participant participated in either the first round or the second round, but not in both rounds.

Prior Experiences with the Word Interface

The participants of the usability study reported candidly about their prior experiences with the Word interface in regard to track changes. They reported having issues interacting with the Word ribbon menu, finding document revisions and comments, and reading issues with the JAWS screen reader.

The participants commented on their inability to properly interact with the Word ribbon menu (Figure 1). They commented, “I never understood the track change and the ribbon is difficult,” and “Some of the menu options do not read properly with JAWS.”

Figure 1. Word ribbon menu.

In order to gain access to the revisions and comments contained within a Word document, the ribbon menu (Figure 1) has to be accessed and navigated. Direct access is available by utilizing keyboard hot keys such as Alt r, but this requires memorization and could lead to cognitive overload. Alternatively, the ribbon menu could be manually navigated by pressing the keyboard Tab key. When using the Tab key, users hear each ribbon menu element as the screen reader presented the content audibly. This option can be monotonous.

Persons who are blind do attempt to utilize Word’s track changes feature for reviewing revisions and comments contained within a Word document because they encounter difficulties. One participant commented, “I have tried to use track changes from time-to-time. I had problems selecting and finding items.” Another participant commented, “It is difficult to find comments and determine what changes others have made.”

Even when the revisions and comments are found, the revisions are difficult to interpret. This is often more difficult when both a deletion and an insertion occur in the same sentence (Figure 2). Interpreting this content is difficult to comprehend and may require the content being repeated several times. Persons who are blind usually give up in frustration and accept the revisions, just to be able to listen to a clean sentence containing no revisions or comments.

Figure 2. Example of two types of revisions (deletion and insertion) occurring in the same sentence.

Another difficulty reported, while reviewing revisions and comments, is the inability to distinguish the comments made by a co-author from the paper content. For example, for the comments presented in Figure 3, the JAWS screen reader reads “… but in a manner to be prescribed by law comment sj1 John’s Comment.” Thus, causing confusion on the part of a person who is blind as to whether “sj1 John’s Comment” is the content of the document or is a comment made by a co-author.

Figure 3. Word comment.

Therefore, given the difficulties with the access and use of the ribbon menu to access revisions and comments and the difficulties of interpreting the context and content of revisions and comments, the Word add-in prototype evolved to include Windows message boxes to present revisions and comments so the JAWS screen reader could properly read and audibly present their context to persons who are blind.

The next sections present the two rounds of the study including the designs for Prototype 1 used in the first round of usability testing and Prototype 2 used in the second round of usability testing.

Prototype 1 Design

Word was selected as the basis for the developed prototype because of its popularity and use in office and collegiate environments. We developed a Word add-in (Prototype 1, CollaborateWithWord) to improve the accessibility and usability of the collaborative writing features (such as searching, accepting, and rejecting a revision or comment) without having to reinvent the wheel for all other editing features. The add-in included options to the standard Word command bar as its primary interface for the participants to initiate the collaborative review process (Figure 4).

Figure 4. Options provided by Prototype 1 are presented inside the red box.

To display all the possible options provided by Prototype 1, a participant performed either a right-mouse click or pressed the menu or application key on the keyboard when the cursor was positioned anywhere on the Word document. All the features provided by Prototype 1 were displayed under “Additional Actions” in the drop-down menu (see the red box in Figure 4). The participant reviewed each option by pressing the up or down arrow keys to navigate the list. As each option in the list was selected as the current focus, the JAWS screen reader audibly read the text of the option to the participant. The participant selected a specific menu option by pressing the keyboard Enter key.

Two levels of context of the revisions or comments were provided in the prototype: paragraph context or sentence context. The steps to review both contexts for revisions and comments within a Word document were identical except for the amount of contextual information provided for each revision or comment. The following example is based on the paragraph context level:

1. Participants started the review of the revisions or comments within a document by selecting the “Paragraph Context of Revisions/Comments” option (the first element in the red box in Figure 4).
2. Once the “Paragraph Context of Revisions/Comments” option was selected, an overview message box (Figure 5) was presented. This message presented structural information including the number of paragraphs, revisions, and comments contained in the document being examined. This provided the participants with a very convenient way to access the status of the document they were reviewing.
3. The review of the document content and its associated revisions and comments began once the “OK” command button in the message box was selected. If the “Cancel” command button was selected, the prototype was exited and the review process was terminated.

Figure 5. Overview message box showing the structural information of a document.

For each of the message boxes presented, the JAWS screen reader audibly presented the content of the message box as well as the text of the command buttons as they were accessed. The participant accessed the command buttons by pressing the Tab key on the keyboard.

Detailed Design of Accepting or Rejecting a Revision or a Comment in Prototype 1

The process of reviewing revisions and comments within a document began by presenting the very first paragraph containing a revision or comment (Figure 6) after the participant pressed “OK” when reviewing the structural information (Figure 5). Pressing the Enter key activated the “OK” command button in the message box and moved on to the next step.

Figure 6. Paragraph message box containing a revision or comment.

The prototype automatically highlighted the corresponding revision or comment contained in the paragraph (Figure 7). The JAWS screen reader audibly presented the highlighted text to the participant. In this case, “well regulated” was audibly presented as an insertion to the document. At this point, the participant chose to either accept or reject this insertion.

Figure 7. A selected revision in review is highlighted.

The participant accessed the Word ribbon menu to accept or reject the proposed revision (insertion of “well-regulated” in the above example). Pressing the Alt r key combination accessed the “Review” Microsoft ribbon menu (Figure 8). The JAWS screen reader read the “Review” tab and audibly presented “Review” to the participant.

Figure 8. Word “Review” ribbon menu.

The participant pressed the A key for the “Accept” drop-down menu or pressed the J key for the “Reject” drop-down menu.In the screen shot presented, the participant pressed the A key to present the “Accept” drop-down menu (Figure 9). The JAWS screen reader read the “Accept” drop-down options list and audibly presented “Accept” to the participant. The participant could then press the keyboard up and down arrows to access and hear each item in the options list. The JAWS screen reader audibly read the following options in succession as the down arrow key was pressed: Accept and Move to Next, Accept Change, and Accept All Changes in Document. In this case, the participant selected the “Accept Change” list item to accept the inserted text revision (Figure 9).

Figure 9. Word “Accept” ribbon menu option list.

Word presented the final outcome once a revision was accepted or rejected (Figure 10). In this case, the result of accepting the inserted text was presented in the document. Figure 10. Word document after the insertion of “well-regulated” was accepted.

Prototype 1 Usability Test Results

Five participants who are blind independently examined the Word add-in Prototype 1. All of the participants completed all tasks utilizing this prototype. When comparing to the 62% task completion rate of the participants of the baseline usability study utilizing the standard Word interface (Schoeberlein & Wang, 2012), this finding provides some evidence that Prototype 1 did improve participant performance for task completion associated with accepting and rejecting revisions and comments within a Word document over the traditional Word interface.

During the usability study sessions, the begin time and end time was recorded for each task in the task list. The recorded time was utilized to determine a time range that could be quantified as a task completion time. The average time spent on completing each task by the participants is presented in Table 1.  

Table 1. Prototype 1 Usability Study Task Performance

The participants were able to tell whether the document they were dealing with was a final version or a document containing revisions. They were especially pleased to see that “the revision or comment text appears much easier to distinguish from existing text than when using the word features themselves.” However, they complained about the reliance on the Word ribbon menu for accepting or rejecting revisions and comments due to the difficulty of accessing the ribbon menu. Even though most of the participants could be considered “medium to power/expert” computer users, it took them a long time to traverse the options in the ribbon menu or to figure out how to accept or reject a change. In addition, the participants asked for a mechanism to repeat a revision or comment in case either the context or the actual revision or comment was not understood during the first round of review or attempt.

Twenty-four category cases were identified from the audio recording transcription. A Cohen’s Kappa measure of 100% verified that there was evidence of acceptable agreement and reliability between the coders. Two main categories were identified including request for change and issues (Table 2).

Table 2. Prototype 1 Audio Transcript Coding Category and Participant Comment Examples

Although all the participants were able to complete the tasks, it was very important to address the issues raised. Therefore, a second round of design and usability testing was conducted based on the comments of the participants from the first round.

Prototype 2 Design

The design of Prototype 2 focused on the removal of the reliance on the Word ribbon menu for accepting or rejecting revisions. The JAWS screen reader still read the message box text and audibly presented the text to the participant.

Overall Design Comparison Between Prototype 1 and Prototype 2

In Table 3, we present a comparison between Prototype 1 and Prototype 2 for accepting a revision. The Prototype 1 design included five steps to accept a revision and interaction requiring the use of the Word ribbon menu. In addition, Prototype 1 needed to be restarted when the participant wanted to move on to review the next available revision or comment. The Prototype 2 design included three steps to accept a revision (steps 1 to 3a) without requiring the interaction with the Word ribbon menu. The additional steps (steps 3b to 3c) allowed for the rejection of the revision and the repeating of the review of the revision, and then automatically advanced to the review of the next available revision or comment. All message boxes content and command buttons text were audibly and visually presented, allowing both visually able persons and persons who are blind to interact with each Prototype.

Table 3. Design Comparison Between Prototype 1 and Prototype 2

Detailed Design of Accepting or Rejecting a Revision or a Comment in Prototype 2

Instead of relying on the complexities of the ribbon menu for accepting or rejecting revisions and comments, Prototype 2 used Windows message boxes as an alternative approach to present questions in order to determine acceptance or rejection of a particular revision or comment. The participant simply responded to a question with a “Yes” or “No” (Figure 11) before moving on to the next step. Separate message boxes were used to present either the acceptance or rejection of a revision or comment. The procedural steps of the process to accept or reject a revision or comment were further enforced by the Prototype 2 design.

Figure 11. Accept revision message box.

The “Yes” or “No” response was determined by which command button the participant selected with the keyboard. For persons who are blind, they simply pressed the Tab key to toggle between the “Yes” or “No” command buttons. The JAWS screen reader read the text of the selected command button and audibly presented the text to the participant. This approach was an improvement over the original reliance on the ribbon menu because there was no need for the participant to memorize special keyboard key presses, such as Alt r, in order to navigate the ribbon menu to accept or reject a revision or comment.

The enhancement that added the ability to repeat a revision or comment was implemented in the same manner to avoid the use of the ribbon menu with the use of message boxes instead. In order to implement the feature of repeating a revision or comment, a message box proposed the question “Do you want to repeat the revision?” (Figure 12).

Figure 12. Repeat revision message box.

In addition to the requested enhancements, to enable participants to go through the whole review process without having to access the ribbon menu, the presentation of the contents of the revision or comment to the participant was included in Prototype 2 (Figure 13). The message box presented the revision number, the type of revision, and the revision content. Figure 13, in this case, shows that the revision in this paragraph is revision number 1 and is inserted text.

Figure 13. Insert revision message box shows the revision number, type of revision, and content of the revision.

Prototype 2 Usability Test Results

Six participants who are blind independently examined the Prototype 2 in this round. None of the participants in this round participated in the Prototype 1 test—they were all new to the prototype. The same procedures for Prototype 1 usability sessions were also used for the Prototype 2 sessions.

Similar to the Prototype 1 usability test, all participants of this round of the study were able to complete all the assigned tasks. Average task completion time for this round is presented in Table 4.

Thirty-six category cases were identified from the transcripts for this round. The majority of issues were focused on the interaction mechanism used in the prototype, with some suggestions on how to further improve the prototype (Table 5). No additional issues were identified. The coding from both coders had a Cohen’s Kappa of 100%, providing evidence of acceptable agreement and reliability between the coders.

Table 4. Prototype 2 Usability Study Task Performance

Table 5. Prototype 2 Audio Transcript Coding Category and Participant Comment Examples

When the participants were asked to give comments on their experience while interacting with Prototype 2, they noted, “the dialogs [in the Prototype 2] follow the Windows usual dialogs”; it is “much easier to locate track changes and make corrections if needed. Also, it makes me feel even more independent when it comes to doing my work without calling on a sighted person to do the track changes for me”; and “It was simple to use it once I understood how it worked, but the best part was that it only took a few minutes to fully understand how to use this program.”

Performance Comparison Between Prototype 1 and Prototype 2

The following sections discuss comparisons between the two prototypes for the task completion time and the perceptions of each prototype.

Comparison on Task Completion Time

The tasks performed during the usability study included searching for all revisions, accepting a revision, rejecting a revision, and rejecting a remaining revision. As discussed in previous sections, when comparing the design of the two prototypes, in order to remove the reliance on the ribbon menu, we used message boxes to force the participants to make choices in terms of whether to accept, reject, or repeat the review of a revision. Even though the participants did not need to switch between the two interfaces, they did need to take some extra time to respond to the questions raised. Therefore, we anticipated no difference in terms of the time needed to complete the assigned tasks. Hence, we have the following hypothesis: There is no difference in the task completion time for the assigned tasks. The following are sub-hypotheses to this main hypothesis.

• H0a: There is no difference in the task completion time for searching for all revisions using Prototype 1 versus using Prototype 2.
• H0b: There is no difference in the task completion time for accepting a specific revision using Prototype 1 versus using Prototype 2.
• H0c: There is no difference in the task completion time for rejecting a specific revision using Prototype 1 versus using Prototype 2.
• H0d: There is no difference in the task completion time for accepting or rejecting a remaining revision using Prototype 1 versus using Prototype 2.

Figure 14 presents the average task completion time for the two rounds of the usability study using Prototype 1 in the first round of usability testing and Prototype 2 in the second round of usability testing. The figure shows that the participants using Prototype 2 (the second round) seemed to have spent less time on average to complete the search for all revisions task comparing to the ones using Prototype 1 (first round), while spending a slightly higher amount of time accepting a revision and using a similar amount of time when working on the rest of the tasks.

Figure 14. Comparison of average task completion time in seconds between two rounds of usability study.

To further test the hypothesis, statistical analysis on the data need to be conducted. However, due to the limited samples available and the non-normally distributed data, a non-parametric test, the Wilcoxon–Mann–Whitney test, was conducted. Table 6 below shows the results of the Wilcoxon-Mann-Whitney test for all the sub-hypotheses. All the sub-hypotheses (H0a through d) are accepted, indicating there is no significant difference in task completion time between the uses of the two prototypes. Further usability testing, with additional participants, will be conducted in the future to provide additional data for more statistical analysis.

Table 6. Wilcoxon-Mann-Whitney Test on Task Completion Time   *n₁ = 6, n₂ = 5, alpha = 0.05

Comparison on the Perception of the Prototypes

The data in Figure 15 shows the percentage of agreement and percentage of disagreement with each question or statement in the post-study questionnaire from the two rounds of the usability study, based on participants’ experiences while interacting with either Prototype 1 or Prototype 2. Responses in the first round relate to Prototype 1 and responses in the second round relate to Prototype 2. Neutral responses from the participants are not presented and comprise the remainder of percentages in Figure 15.

Survey responses of “Strongly Agree” or “Agree” are categorized as “Agree,” and survey responses of “Disagree” or “Strongly Disagree” are categorized as “Disagree.” The only question that received “Disagree” responses was to the statement “the prototype (Prototype 1 or Prototype 2) was flexible.” The participants’ disagreement to this question is not necessarily a negative result, but actually a mere statement of fact. The procedural nature of the prototypes does in fact force the prototypes to behave in a structured manner and thus a non-flexible manner.

Figure 15. Comparison of perception between the two groups of participants (responses in the first round relate to Prototype 1 and responses in the second round relate to Prototype 2).

All of the participants from both rounds agreed that they would use the prototype on a regular basis, the prototype was easy to learn, and the tasks were easy to complete using the prototype. They also all agreed that they were satisfied with their tasks’ completion times and they all completed the tasks without any problems.

When the participants were asked if they felt the prototype improved their performance, 60% of the first round participants agreed that Prototype 1 improved their performance, while 83.3% of the second round participants agreed that Prototype 2 improved their performance. Eighty percent of the first round participants agreed that Prototype 1 was clear and understandable, while 100% of the second round participants agreed that Prototype 2 was clear and understandable. Based on the researcher’s discussion with the participants, the removal of the dependence on the Word ribbon menu in Prototype 2, streamlined the process of searching and retrieving revisions and comments and was the main reason for the improvement of favorable responses in regard to understandability and improved performance.

The only negative responses were in regard to the lack of flexibility of the prototype. When the participants were asked about their opinions in regard to the flexibility of the prototype, 20% of the participants who used Prototype 1 responded that Prototype 1 was not flexible and 16% of the participants who used Prototype 2 responded that Prototype 2 was not flexible. The lack of flexibility responses were not entirely surprising, and even though the participants responded candidly to open-ended survey questions that the prototypes were not flexible, they did acknowledge that the lack of flexibility was due to the procedural nature of the steps needed in order to accept or reject revision and comments. Prototype 2 was designed to present the paragraph or sentence containing a revision or comment, followed by the actual revision or comment, and finished with asking the participant to accept, reject, or repeat the revision or comment. This procedural control could be seen as not being very flexible, but necessary.

Overall the Word add-in prototype was accessible and usable for revisions and comments features of Word, and the participants liked the interface.

Next Steps: Prototype 3 Design and Future Testing

Based on the comments from the participants at the conclusion of the Prototype 2 usability session, additional enhancements were made to Prototype 2’s interface. The enhancements included the inclusion of hot key access to the command bar options (Figure 16), a “Help Menu” (Figure 17), and a “Continue” message box (Figure 18) to continue or exit the examinations of revisions and comments. Previously, the default was to automatically continue to the next revision or comment.

Hot key values were displayed on the Word command bar (Figure 16) and identified by the underscored (_) character. The JAWS screen reader audibly emphasized the underscore value when reading the menu option. Pressing the underscored key selected the menu item. For example, after activating the command bar (Figure 16), a keyboard key press of the letter H or h will access the “Help Menu” message box (Figure 17).

Figure 16. Hot key access.

The “Help Menu” provided the keyboard key-press instructions to access a menu option on the command bar and a brief description of each option’s functionality.

Figure 17. Help Menu.

The “Continue” message box (Figure 18) was added to the procedure for Prototype 2 to enable the participants the ability to continue or exit the examination of revisions and comments. Prior to this enhancement, the procedure for Prototype 2 would automatically continue to the next revision or comment in the document. This enhancement enabled a participant to exit the procedure as desired.

Figure 18. Continue message box.

The review process for Prototype 3, with the addition of the “Continue” message box (Figure 18), includes the following steps:

1. Present the “Paragraph Context” message box (Figure 6).
2. Present the “Revision” message box (Figure 13).
3. Present the “Accept Revision” message box (Figure 11).
4. Present the “Reject Revision” message box (similar to Figure 11, except “Reject” instead of “Accept”) if the user selected “No” during step 3.
5. Present the “Repeat Revision” message box (Figure 12) if the user selected “No” during step 4.
6. Present the “Continue” message box (Figure 18).

Once the design and the implementation of Prototype 3 is completed, a longitudinal study will be conducted to properly evaluate and improve Prototype 3.

Discussion

Based on the results of previous studies (Schoeberlein & Wang, 2012), we were able to determine that the interface had to be compatible with the standard keyboard and the JAWS screen reader because persons who are blind utilize these almost exclusively to interact with electronic computer interfaces. In addition to the hardware (keyboard) and software (JAWS) interfaces utilized for persons who are blind, we verified that our interface components, Windows message boxes and Word command bars, were compatible with the JAWS screen reader.

Once we understood the general design components we had available, we proposed features for a prototype for reviewing revisions and comments in a Word document (Schoeberlein & Wang, 2013). The prototypes provided access to the revisions and comments of a document in a format that included the paragraph or sentence context, making the revisions and comments easier to understand. Both rounds of the usability study showed that all participants were able to complete the tasks.

When the task completion times of the participants from the two rounds of the usability study were compared, there was no significance found between the two. However, all of the participants using Prototype 2 reported that they perceived the prototype they used as being clear and understandable, while not all the participants using Prototype 1 thought so. Because Prototype 1 required participants to switch between the ribbon menu and the prototype itself in order to work on any revisions, it may have caused some confusion for participants even though the contexts of the revisions were presented. As for Prototype 2, it implemented a mechanism to guide participants to review each revision by requesting a participant’s input to make decisions step by step. This mechanism allowed the participants to complete all the reviews of the revisions using the prototype rather than switching to the ribbon menu. Therefore, it may have eliminated the possible confusion caused by the Prototype 1 design. As a tradeoff, using Prototype 2, participants may have needed to spend some time to indicate their decisions along the way, which resulted in the similar amounts of time needed to complete a given task.

Even though the prototypes were specific to Word, providing different levels of context (paragraph or sentence) to improve the participants’ understanding of the revisions or comments and providing accessible and usable interface constructs, such as Windows message boxes and command bars, can be applied to improve access and use to any application. Additionally, it is important to consider an add-in for applications that enhances the interface’s functionality.

Implications

Providing the context of revisions and comments is critical to improved collaborative writing. The participants were able to understand the context of the revisions and comments, thus improving their awareness of the specific changes made to a document. Extracting the revisions and comments from the original text made the revisions and comments easily distinguishable, improving review and understandability.

The use of a context-sensitive menu, in the form of a Word command bar, enabled direct access for accepting or rejecting changes, finding the next change, and finding the next comment. The command bar enabled easy access to the collaborative writing features associated with track changes and could be accessed by simply keyboard right-clicking anywhere on the open document.

Utilizing this interface design, with accessible Windows message boxes, the participants thought that the tasks were easy to complete, that they were satisfied with the time they took to complete the tasks and that they were able to complete the tasks without any problems. It is promising that utilizing message boxes to present accessible paragraph or sentence context of a revision or comment and the actual content of the revisions and comments will help persons who are blind participate independently in collaborative writing.

Conclusion

As the answer to our research question, How a collaborative writing interface should be designed in order to be accessible and usable for persons who are blind?, two collaborative writing prototypes were designed and tested using the iterative design approach. These prototypes provided access to the revisions and comments of a document in a format that included the paragraph or sentence context, making the revisions and comments easier to understand. Both prototypes were considered accessible and usable to the participants of the usability study. All the participants considered the interfaces of the prototypes as being clear and understandable, easy to use, improved their performance, and enabled the tasks to be completed without any problems. The participants were satisfied with the time they took to complete the study tasks, and they would utilize the prototypes in the future on a regular basis. However, consider the limited number of participants, continued research and development are needed to improve the design.

Acknowledgements

We would like to thank the National Federation of the Blind (NFB) membership for their support and participation.

References

Buzzi, M. C., Buzzi, M., Leporini, B., Mori, G., & Penichet, V. (2010). Accessing Google Docs via screen reader. Lecture Notes in Computer Science, 6179, pp. 92–99. Chen, X., Tremaine, M., Lutz, R.,

Chung, J., & Lacsina, P. (2006). AudioBrowser: A mobile browsable information access for the visually impaired. Universal Access in the Information Society, 5(1), pp. 4–22.

Dorigo, M., Harriehausen-Mühlbauer, B., Stengel, I., & Dowland, P. S. (2011). Survey: Improving document accessibility from the blind and visually impaired user’s point of view. Lecture Notes in Computer Science, 6768, pp.129–135.

Hampel, T., Keil-Slawik, R., Claassen B. G., Plohmann, F., & Reimann, C. (1999). Pragmatic solutions for better integration of the visually impaired in virtual communities. GROUP ’99 Proceedings of the international ACM SIGGROUP conference on Supporting group work (pp. 258–266). Phoenix, AZ: ACM.

Hermsdorf, D., Gappa, H., & Pieper, M. (1998). Webadapter: A prototype of a WWW-browser with special needs adaptations. Proceedings of ICCHP 98 (pp. 151–160).

Lazar, J., Beere P., Greenidge K., & Nagappa Y. (2003). Web accessibility in the Mid-Atlantic United States: A study of 50 homepages. Universal Access in the Information Society, 2(4), 331–341.

Lazar, J., Dudley-Sponaugle, A., & Greenidge, K.-D. (2004). Improving web accessibility: A study of webmaster perceptions. Computers and Human Behavior, 20(2), pp. 268–288.

Mori, G., Buzzi, M.C., Buzzi, M., Leporini, B., & Penichet, V. (2011). Collaborative editing for all: The Google Docs example, Lecture Notes in Computer Science, 6768, pp.165–174.

National Federation of the Blind (2014). Technology Resource List, Retrieved on August 10, 2014 from https://nfb.org/technology-resource-list

Pitt, I. J., & Edwards, A. D. N. (1996). Improving the usability of speech-based interfaces for blind users. Proceedings of the Second Annual ACM Conference on Assistive Technologies (pp. 124–130). Vancouver, BC, Canada: ACM Press.

Schoeberlein, J. G., & Wang, Y. (2012). Accessible collaborative writing for persons who are blind: A usability study. Proceedings of ASSETS’12 Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility (pp. 267–268). Boulder, Colorado: ACM.

Schoeberlein, J. G., & Wang, Y. (2013). Providing an accessible track changes feature for persons who are blind. In C. Stephanidis & M. Antona (Eds.), Universal Access in Human-Computer Interaction compilation of Lecture Notes in Computer Science (pp. 389–398). Berlin Heidelberg: Springer-Verlag.

United States Government Archives (2011). Bill of Rights. Accessed on January 5, 2011 from http://www.archives.gov/exhibits/charters/bill_of_rights_transcript.html

WebAIM (2014). Screen reader user surveys. Retrieved on August 10, 2014 from http://webaim.org/projects

World Health Organization (2013). Visual impairment and blindness fact sheet. Retrieved on May 15, 2013 from http://www.who.int/mediacentre/factsheets/fs282/en/

Published in: in Volume 10, Issue 1,