Abstract
It is vitally important to coordinate resources, information sharing, and two-way communication between medical incident commanders (ICs) and first medical responders (paramedics) at mass casualty incidents (MCI) sites. Information at the time of disasters also needs to be effectively analyzed and presented through intelligent user interfaces. Such interfaces need to be easy-to-use by ICs to foster critical decisions that can potentially reduce mortality rates. In this paper, we present Panacea’s Cloud™—a next-generation multiple casualty management system. This system has been iteratively developed and refined based on user experience research driven methodology that employed a mixed methods approach, including the views of clinical experts. Panacea’s Cloud™ is an example of a next generation MCI system that has an intelligent dashboard that integrates Internet-of-Things (IoT) technologies such as wearable devices and augmented reality technology (AR), virtual beacons, and sensor network nodes. It supports coordination between ICs and paramedics. Our research demonstrates how IoT-based web applications, especially AR and the use of smart glasses, can be futuristically designed for purposes of smart healthcare applications that have effective and efficient communication capabilities. General design recommendations for next generation multiple casualty management system development include incorporating Situational Awareness features, a Synchronous Map View system, a Hands-Free Communication service with AR and smart glasses, Digital Notes, and resilient Wi-Fi networks.
Keywords
next-generation incident management system, UX research for design refinement, eye tracking, disaster management systems, IoT-based web application, augmented reality, smart glasses, triage, efficiency, effectiveness, satisfaction
Introduction
Natural incidents such as hurricanes and floods as well as man-made disasters, for example car collisions and building collapses, can result in mass casualties. Such disasters require a coordinated medical triage response under time-intensive and often limited resource situations (Russo, Galante, Jacoby, & Shatz, 2015). Pertinent communication among healthcare professionals at mass casualty incident (MCI) sites is critical for effective and efficient disaster response management. The healthcare professionals involved in these events usually include medical incident commanders (ICs) and first medical responders (paramedics) who are tasked with the triage of patients and for rendering resource allocation decisions that prevent under triage (i.e., inadequate care that may lead to more serious health conditions) or over triage (e.g., unwarranted care efforts that lead to increased/unnecessary healthcare costs). Thus, the fundamental aim of employing an effective management system during disaster is to reduce the mortality rate, contain healthcare costs, and prioritize the evacuation of victims (Aylwin et al., 2006).
When MCIs occur, the triage status of patients is of high relevance to healthcare professionals and many other stakeholders, for example, hospitals, facilities, medical supplies, government administrators, military, police, and fire departments (Aguiar, Vieira, Galy-Marie, & Santoni, 2015; Demir, 2015; Kattimani, Tiwari, Pandi, Meka, & Lingamaneni, 2015). To coordinate the different professionals, trusted sources and systems that are dynamic and able to customize responses rapidly are needed to provide real-time information. New intelligent user interfaces promise to increase the coordination quality and reduce the time frame to do this. Lee, Wang, Wang, and Cheung (2012) and Ferguson, Gesing, and Nabrzyski (2016) described the needs for such tools in order to promote the ability to make better-informed decisions (Lixin, Lingling, Dong, Junxue, & Zhanwu, 2011).
To address this need, a next-generation MCI command system named, Panacea’s Cloud™ was conceived and developed. We initially began by identifying the challenges users face today in working with existing systems to handle MCIs (EMS1, n. d.). Based on the identification of the stakeholder requirements for improving MCI ability to cope with triage scenarios, Panacea’s Cloud™ was designed and developed with the aim of addressing users’ needs. The IT-solution features an infrastructure-independent platform with an intelligent dashboard designed to integrate Internet-of-Things (IoT) technologies such as wearable devices, virtual beacons, and sensor network nodes. The system supports coordination between the ICs as dashboard users and paramedics at MCI sites. It utilizes augmented reality (AR) technology that supports the production of a live direct view of real-world environments whose elements are augmented by technologies such as videos, graphs, or GPS data (Liao, 2016). The intelligent dashboard has the capability of tracking real-time patient status, responder status, and supply status through a context-based geolocation service system (Salman, Cheng, & Patterson, 2012).
In this article, we describe our research concerning Panacea’s Cloud™ user experiences, as well as our study methods and results. The results show that the new system can provide improved communication and MCI coordination opportunities. General design recommendations for next generation MCI system development include improving Situational Awareness systems, Synchronous Map Views, Hands-Free Communication ability, smart glasses, Digital Notes, and resilient Wi-Fi networks.
Literature Review and Related Work
Central problems that negatively impact ICs and first responders’ performance have been discussed in previous studies. Mentler and Herczeg (2015) illustrated the challenge of group dynamics, the user’s high cognitive workload, and organizational working conditions (read also Reddy et al., 2009).
Previous studies also found existing systems to have inadequate approaches to information collection, limited information visualization ability, and ineffective patient triaging. For instance, Jiang et al. (2016) showed that 35% of the responders viewed communication as the major stumbling block in influencing responses during mass casualty incidents; 46% of responders reported problems with the lack of information or necessary resources. Among paramedics who are usually equipped with handheld radios, 70% reported that radio communication was the most frequently used technology to track and care for patients during a mass casualty incident, but only 15% said they had the capability of tracking care status in real time (Jiang et al., 2016). Responders’ experience can help incident commanders with their decision making process and performance (Aguiar et al., 2015). This is crucial for rendering better-informed decisions during MCI rescue operations (Lixin, Lingling, Dong, Junxue, & Zhanwu, 2011), especially where the lack of optimal communication, via radios only, might cause tragic results.
Mentler and Herczeg (2015) reported that physicians took notes on notepaper or other paper-based formats, which often delayed results and increased failure rates because of poor handwriting and inadequate data. An indoor-fire case study revealed that the unreadable handwritten forms used to register patients and staff prompted a delay in hospital services (Koning, Ellerbroek, & Leenen, 2014). Therefore, a requirement for any new MCI system is a mechanism for digital note taking and digital forms for collecting consistent information.
Dynamic data and map presence are also critically important factors. Static displays and mouse‑based interaction techniques mostly fail to provide information on the dynamic environment. Results by Ley et al. (2014) revealed that users who coordinate emergency responses desire synchronous maps, an annotation tool and a form of technology support, that allows them the real-time ability to view the paramedics’ actions in the field as these are occurring. Dynamic real-time information approaches, large displays, and tangible interaction techniques may, hence, promote more promising MCI solutions than those attainable at present.
Incident commanders report wanting to track first responders live geographic locations with the system, but note that poorly designed user interfaces may cause obstacles. In this regard, Paelke et al. (2012) showed that improvements in disaster management applications require improvements of real-time tracking of equipment and rescue staff, and more effective visual representations of the situation. Geographical information on an interactive map can help ICs to make decisions as quickly as possible (Doeweling, Tahiri, Sowinski, Schmidt, & Khalilbeigi, 2013) and can visually inform ICs to identify first responders who are closer to the incidents (Park, Cullen, & Smith-Jackson, 2014). Integration of real-time views through smart glasses can be of great value for geographically distributed emergency managers and responders to the scene (Blum, Eichhorn, Smith, Sterle-Contala, & Cooperstock, 2013).
The lack of information visualization is considered a further problem for MCI systems (Ellebrecht, Feldmeier, & Kaufmann, 2013). Research reveals that existing visualization solutions do not provide full coverage, but only partial coverage of the incident (Ellebrecht et al., 2013). Many emergency systems use a tremendous number of icons on displays that may confuse users and cause increased cognitive workload (Salman, Cheng, & Patterson, 2012). From a design perspective, it was suggested that the icons of medical management systems should be differentiated (Salman et al., 2012).
Patient triage is another problem addressed in previous MCI management research. Patient triage informs rescue staff of priority transportation to the hospital for the most critically injured victims. Results show that digital triaging of patients reduces 43% of the evacuation time when compared to paper-based triage systems and also reduces the stress level of on-scene personnel by providing control of the incident electronically (Ganz et al., 2014).
Usability Issues in MCI Management Systems
Usability research plays a significant role in the effectiveness and operability as well as satisfaction and performance of disaster management systems (Blomqvist, 2014; Meissner, Luckenbach, Risse, Kirste, & Kirchner, 2002).
Usability challenges of existing systems include, but are not limited to (a) efficient navigation, (b) data visualization, and (c) the fast browsing of datasets (Mentler & Herczeg, 2015).
The MCI system’s goal is to speed up the rescue process. This can be achieved by utilizing a user interface design that reduces distraction, prioritizes tasks, creates flexible workflows, avoids time delay, and supports improved intuitive interaction (Artinger et al., 2012). Interactions can be improved with head-mounted displays that allow responders to free their hands when alone during the patient triage process or with hazardous materials. Such displays also can improve situational awareness and consequently can increase co-ordination quality (Vassell, Apperson, Calyam, Gillis, & Ahmad, 2016).
In creating a user-friendly interface design, all types of users should be involved in order to obtain the best results (Berndt, Mentler, & Herczeg, 2016). ICs need more user-friendly applications that offer the right information when required, rather than having to create more information (May, Mitchell, & Piper, 2014). Also, an optimal visual overview can improve the analysis of a current situation and IC usability and outcomes (Danielsson & Alm, 2012).
Existing research stresses the importance of IT systems for purposes of communication and coordination in MCIs and disaster management situations. What the existing systems lack is a combination of the following key features: (a) real-time geographical positioning of patients and paramedics, (b) a dynamic easy to view map, and (c) built-in audio/visual conferencing with hands-free technology, for example, wearable technologies such as Google Glass or Recon Jet™. Panacea’s Cloud™ is a first prototype that tries to include all these missing functionalities by using Recon Jet for communication and coordination purposes. We illustrate the new system in the following section as well as results of our user experience research.
Panacea’s Cloud™ Technology
Panacea’s Cloud™, a web-based technology device that supports communication between medical incident commanders (ICs) and medical first responders, includes smart glasses that can be used by first responders/paramedics.
The technical features of Panacea’s Cloud™ include a mobile infrastructure that uses a custom enclosure including a combination of a mobile router, a Raspberry Pi server with the corresponding database, and a rugged weather-proof box to protect the components. Panacea’s Cloud™ is composed of a network that runs independent of Wi-Fi and other external networks. This fault-tolerant network and mobile cloud allows for deployment anywhere and eliminates failures. Once paramedics switch the enclosure on, the system provides a Wi-Fi signal for their smart glasses to connect to the system. Using a chain of such devices, the network can be connected to the Internet grid and to the IC’s remote dashboard. The smart glasses are able to establish audio/video communication and provide real-time location data using the built-in GPS.
Panacea’s Cloud™ consists of a dashboard screen for ICs to use and is available on any computer connected to the Internet. The paramedics wear a heads-up display (i.e., Recon Jet Smart Glasses) where each unit has an integrated GPS system that determines geographical location as well as the Wi-Fi capability of establishing a live audio/video communication with the IC (Figure 1). Through its wireless networking and mobile cloud technology, the system allows ICs to better manage patients’ medical treatment than other technologies. (Gillis et al., 2015).
Figure 1. Panacea’s Cloud™ provides audio/visual communication between paramedics and medical incident commanders using its own Wi-Fi technology.
The dashboard (Figure 2) allows ICs to map first responders and to establish audio/visual communication with them. The interactive dashboard map displays location data and the status of each patient, and the IC is able to create and store an incident report, by using the Panacea’s Cloud™ system. An IC can also add or edit patient information and monitor available hospital supplies in the area, as well as assisting in patient triage efforts.
Figure 2. Panacea’s Cloud™ dashboard view.
The mobile beacons of the system are secured by wrist straps placed on patients in order to track their location. These virtual beacons allow the ICs to track all patient triage events in real time on the dashboard by using a Wi-Fi network.
Panacea’s Cloud™ also permits the use of a hands-free, audio/visual communication system between paramedics and ICs by employing a smart glass system. The hands-free communication feature using smart glasses differentiates Panacea’s Cloud™ from other existing systems. It thus facilitates real-time triage and permits more situational awareness than other systems that can aid in making more informed decisions.
Methods and Iterations
We studied Panacea’s Cloud™ by utilizing usability and user experience methods during two iterations. We conducted the study in the Spring of 2016 and applied mixed methods in order to study its effectiveness, efficiency, and the users’ satisfaction with the technology. In the following sections, we describe the methods we employed (Figure 3).
Figure 3. Study phases and methods.
In Phase 1 we included a design review. After that, the prototype was revised. The improved prototype was then tested in an eye-tracking study and by conducting a usability evaluation, a SUS survey, and a set of topical expert interviews (Phase 2 and 3). The results pointed to agreement about the tool as being user-friendly, but also yielded recommendations for improvements before launching the system in practice.
Iteration 1: Design Review (Phase 1)
The design review was conducted by two medical experts and three usability experts. Its goal was to provide feedback for helping the developers create a more resilient system based on the research findings. In order to achieve this goal, two trauma center surgeons reviewed Panacea’s Cloud™ to identify the gaps between the systems’ design and real-world issues. The medical expert review focused on Panacea’s Cloud™ dashboard, and the experts evaluated the content, functionality, colors, symbols, and labeling of the system. After Panacea’s Cloud™ dashboard was redesigned, the same experts reviewed the system once again. Additionally, three usability experts conducted a design review that was inspired by Nielsen’s Heuristics (Nielsen, 1994). Additional recommendations for improvements were then provided.
Iteration 2: Usability Evaluation with the Eye Tracking (Phases 2 and 3)
In the second phase of the research, we focused mainly on task completion rates (effectiveness) and time on tasks (efficiency). Thus, we created a disaster scenario derived from the major tasks at the time of a disaster. The participants were given no time-limit tasks.
A total of 10 users conducted a usability test using eye-tracking methods. Eye-tracking technology was used to measure efficiency and effectiveness which helped the researchers to better understand what users were looking at and how much time they spent while browsing and completing the tasks by recording the users’ eye gaze movements (Demir, Karakaya, & Tosun, 2012). We analyzed (a) task completion success (effectiveness), (b) task completion time (efficiency), (c) total visit time by page and by areas of interest (AOI), (d) participants’ browsing patterns, and (e) heat maps of the index page. The remote eye tracker was a Tobii system installed on a Dell brand laptop. Stimuli were presented on a 14.4-inch monitor at a resolution of 1024 x 768 pixels and with a refresh rate of 60 Hz.
Think aloud
The task completion test involved the observation of participants’ interactions with the computer during their tasks and the collection of their think-aloud comments about the system (Boren & Ramey, 2000; Nielsen, 1992). We studied participants’ preferences, how they dealt with the system, and opinions (Nielsen, Clemmensen, & Yssing, 2002). Participants were asked to describe their impressions and experiences with the Panacea’s Cloud™. Participants were asked to comment on difficulties that they faced when completing the tasks and were asked to offer suggestions regarding the design and functionality of the system.
Scenario for testing: Description of the six tasks
The scenario focused on a tornado disaster. Task 1 required participants to become familiar with Panacea’s Cloud™ and share their initial impressions. Task 2 focused on the role of the physician at the University Hospital. The scenario presented a situation in which a tornado caused the roof to collapse in a nearby area. The instruction was “You want to communicate with paramedics in the field and survey the emergency situation. Please check the current status of the incident.” Then, users needed to find the paramedics at a specific location (Task 3) and coordinate with them to answer the following questions (Task 4): “How many injured are at the Hall?” and “Who needs immediate treatment?” They were asked to add patient information into the system. Task 5 informed the paramedics that one patient had a change in her health condition, and users were asked to make changes (such as change the patient’s condition from “delayed” to “immediate”). Task 6 focused on the report to the Director of the hospital (e.g., the number of patients that needed immediate attention in the nearby area).
At the beginning of each session, a written task statement was displayed on the screen for the participants. Then participants started the task by pressing the space bar to load the associated system features. The eye-tracking system marked the start and end times for each task. After completing one task, participants pressed the space bar to move on to the next task and repeated the procedures until all tasks were completed. To complete Task 4, participants were asked to use the video call feature to contact paramedics. One graduate assistant from the research team acted as a paramedic by using a live connection applied to a pair of Recon Jet smart glasses to represent the paramedic’s view while providing images of patient injuries. Paramedics reported the information that included the patients’ name, age, gender, and current status. That patient information was provided along with varying patient pictures illustrating the status of patients along with the patient condition (e.g., 40-year-old male, severe lacerations and bruising on right side of chest area with significant pain; vital signs: BP: 130/84, P: 110 and irregular, R: 16 and shallow, SPO2 90%). Additionally, the sounds that the participants made (e.g., talking, sighing, etc.) and mouse actions were captured as video files. Eye gaze movements were captured by a Tobii system.
Surveys and Interviews
The following sections present information about the types of surveys used, how interviews were conducted, and how participants were selected.
User satisfaction survey (SUS)
We collected data in order to measure users’ satisfaction using the System Usability Scale (SUS; Bangor, Kortum, & Miller, 2008; Lewis & Sauro, 2009). Participants were given the SUS directly after completing the session with the six tasks. SUS is a 10-item Likert scale designed to measure users’ subjective satisfaction level (Lewis & Sauro, 2009). The SUS survey was applied to measure the user’s subjective satisfaction level with the Panacea’s Cloud™ system. SUS has outstanding reliability, validity, and sensitivity to manipulated variables (Sauro & Lewis, 2012). On a range of scores from 0–100 points, most usability researchers accept 68 points as the average of user satisfaction. Higher scores are considered more satisfactory designs (Sauro, 2011a). Although SUS provides an understanding of users’ satisfaction level, a single difficult task may reduce the SUS result by 8% (Sauro, 2011b).
Post-study survey and interview
Following the SUS, a post-study survey was conducted to gather dashboard users’ experiences. It was clustered in two parts. The majority of the items were reported on a 5-point Likert response, which is a widely used item-scoring scheme to quantify participants’ opinions and interests (Bishop & Herron, 2015). Part 1 consisted of demographic variables, for example, age of participants and experience in use of technology. Part 2 included 10 questions regarding participants’ previous experiences during the mass casualty exercise and thoughts about the Panacea’s Cloud’s™ features: for example, Item #4 “The audio/video feeds significantly improve my ability to triage mass casualties over my current communication” and Item #5 “The real-time location mapping of patients and staff significantly improves my ability to triage mass casualties over my current communication methods.” Part 3 included five open interview questions such as final thoughts about the system design, terminology used, and recommendations.
Selection of participants
We invited 15 health professionals from the University of Missouri Hospital’s Level One Trauma Center to participate in this study. A total of 10 users agreed to participate in the study. Inclusion criteria were a range of job experiences, specialization, experience in using digital systems, and experience in handling mass emergency situations.
Four participants had up to five years of practicing medicine, two participants reported their work practices ranged between 11 and 15 years, and four had over 20 years of practice experience. The type of specialization included four physicians and six nurses. The majority worked at hospitals in a mid-western US state. All 10 participants reported that they had worked with electronic and digital medical records. Two of them had done this for up to five years and the other eight for between 6 and 15 years. Two participants reported their technology skills were at the beginner level, four were at the intermediate level, and the other four were at the advanced level. Finally, 6 of the 10 participants reported experience in a mass emergency situation with varying roles such as emergency room physician, nurse, dispatcher, and technician.
The participants signed a consent form before they started the testing.
Results
The following sections present the results for each iteration of the study.
Iteration 1: Design and Expert Review
The design and expert review revealed necessary improvements such as missing features and other features needed to be adjusted. For example, incident commanders were able to see the location of the paramedics on the interactive map, but were not able to identify them directly. The paramedics’ status did not reflect real-time data. The IC, therefore, could not understand whether the paramedic was active or not. In addition, it was difficult for the user to count the total numbers of patients in long listings. There was no sorting function for listed patients by name and other relevant information. Also, locating the incident location by address was challenging for ICs because the location search by address was missing on the homepage.
Based on the recommendations in the first iteration of the research, the software development team added direct call buttons with the paramedics’ name on the map view. The new design used grayed out “Call Me” buttons for inactive paramedics. An auto count total numbers feature was added for long patient listings as well as sorting listings by relevant information. To include information for patients that were being continually added, the patient list was automatically synced to show them on the interactive map. Also, a location search field was created to help ICs locate the incidents by their addresses.
After the development team attended to all of the aforementioned problems, the second iteration began.
Iteration 2: Task Analysis with Eye-Tracking results
A total of 8 out of 10 participants completed the tasks successfully. The first participant didn’t complete Task 4 because of a connection error between the IC and paramedic. Unsuccessful tasks indicate that participants gave up trying to perform the task.
Effectiveness: Task completion success rates
Effectiveness is commonly measured through participants’ use of the product to accomplish tasks successfully. In our study, six tasks were defined and conducted by 10 participants, which means 60 tasks in total were completed. As Table 1 shows, only six of the 60 tasks failed; that was a 90% success rate. In reality, the first participant experienced a connection error with the paramedic because of technical problems. A total of three participants failed on Task 6, which refers to sorting the list of patients who need immediate care. All participants achieved the tasks on checking the incident location and finding paramedics at a specified location (Table 1).
Table 1. System Effectiveness (Task Completion)
Participant |
Task 1 |
Task 2 |
Task 3 |
Task 4 |
Task 5 |
Task 6 |
Mean completion rate |
Get familiar with Panacea |
Check incident at LH |
Find PM at LH |
Connect PM, gather patient info, log them into the system |
Change patient status |
Inform director |
||
P1 |
1 |
1 |
1 |
0 |
1 |
1 |
83 |
P2 |
1 |
1 |
1 |
1 |
0 |
1 |
83 |
P3 |
1 |
1 |
1 |
1 |
1 |
1 |
100 |
P4 |
1 |
1 |
1 |
1 |
1 |
1 |
100 |
P5 |
1 |
1 |
1 |
0 |
1 |
1 |
83 |
P6 |
1 |
1 |
1 |
1 |
1 |
0 |
83 |
P7 |
1 |
1 |
1 |
1 |
1 |
0 |
83 |
P8 |
1 |
1 |
1 |
1 |
1 |
1 |
100 |
P9 |
1 |
1 |
1 |
1 |
1 |
1 |
100 |
P10 |
1 |
1 |
1 |
1 |
1 |
0 |
83 |
Mean total score |
10 |
10 |
10 |
8 |
9 |
7 |
90 |
Note. N = 10; 1 indicates success; 0 indicates failures
The mean of the task completion scores was 90 with a standard deviation of 8.78 and a resulting 95% confidence interval that ranged from about 81 to 96. When we applied Lah and Lewis’ (2016) curved grading scores for interpreting our data, the results show the mean of 90 corresponds to a grade of B-, and the lower and upper limits of a 95% confidence interval around this mean ranged from about 81(C) to 96 (B).
Efficiency: Task completion time
The time it takes to complete a task is critically important particularly in mass causality situations that may directly affect people’s lives. The ICs should accomplish the required tasks quickly and easily even during stressful working conditions. The Panacea’s Cloud™ system dashboard is designed to accomplish all required tasks by ICs. We measured completion time for each task to examine the efficiency of Panacea’s Cloud™ (Table 2).
Table 2. System Efficiency (Task Completion Time, mm: ss)
Participant |
Task 1 |
Task 2 |
Task 3 |
Task 4 |
Task 5 |
Task 6 |
Total time on tasks |
Get familiar with Panacea |
Check incident at LH |
Find PM at LH |
Connect PM, gather patient info, log them into the system |
Change patient status |
Inform director |
||
P1 |
5:38 |
2:27 |
0:25 |
-* |
4:52 |
2:06 |
17:46 |
P2 |
3:45 |
1:40 |
1:31 |
11:28 |
5:25 |
0:51 |
25:06 |
P3 |
3:58 |
1:29 |
0:19 |
6:10 |
0:23 |
1:02 |
13:35 |
P4 |
8:21 |
7:20 |
2:44 |
11:26 |
0:50 |
2:01 |
33:10 |
P5 |
5:19 |
1:28 |
1:05 |
8:25 |
1:38 |
2:29 |
20:40 |
P6 |
4:33 |
2:00 |
0:14 |
16:26 |
0:53 |
2:09 |
26:25 |
P7 |
6:48 |
2:47 |
0:27 |
10:18 |
0:55 |
1:19 |
22:56 |
P8 |
14:28 |
3:34 |
0:41 |
15:37 |
0:31 |
0:57 |
36:20 |
P9 |
1:50 |
3:35 |
1:04 |
6:12 |
0:58 |
0:16 |
14:31 |
P10 |
4:35 |
2:41 |
5:09 |
13:29 |
0:40 |
3:03 |
30:10 |
Average time |
5:55 |
2:54 |
1:21 |
11:03 |
1:42 |
1:37 |
24:03 |
Note. N = 10
* indicates missing data due to technical difficulties
Efficiency results revealed that participants, as the first-time users of the Panacea’s Cloud™ system, completed entire tasks in an average of 24:03 minutes. Task completion time per task ranged from 1:21 minutes to 11:03 minutes. Participants took 5:55 minutes on average to become familiar with the system. Task 4, which consisted of forging an audio/visual connection with paramedics on the scene to identify the situation, to retrieve patient information from paramedic, and to log the patients’ information into the system, took an average of 11:03 minutes.
Total visit duration by page
Participants’ eye gaze movements were tracked and recorded for each task. Total visit time duration by page gives information on which pages participants spent the most time on to complete the tasks. Table 3 indicates users visited the index page for the longest period of time (386 seconds on average). The index page features the interactive map that participants frequently used to accomplish the tasks. After the index page, the second longest time taken on task was on the video feeds page, where participants spent a mean of 248 seconds in total visit time.
Table 3. Total Visit Duration by Page
Mean (minutes) |
Range (quickest to longest in minutes) |
|
Index page |
6.43 |
3.20-11.10 |
Video feeds page |
4.14 |
2.50-8.20 |
Patient list page |
2.44 |
1.38-3.17 |
Incident list page |
2.10 |
1.26-3.05 |
Add patient page |
1.62 |
1.19-2:49 |
Staff page |
1.20 |
0.57-2.13 |
Supplies page |
1.08 |
0.30-1.47 |
Notifications page |
0.48 |
0.13-1.14 |
Add staff page |
0.21 |
0.8-0.53 |
Create incident page |
0.20 |
0.9-1.03 |
Login page |
0.15 |
0.5-0.44 |
Note. N = 10
Total visit duration on index page by AOI
When users attempt to complete a task, they often fixate their gaze upon certain parts of a page. Because the total visit duration time by page (Table 3) clearly demonstrated that the users fixated the longest time on the index page (homepage). The index page has been further divided into six parts, as areas of interest (AOIs), to better understand which part attracted the most visual attention. The pre-defined AOIs on the index (home) page of Panacea’s Cloud™ included (a) Filter Patient, (b) Filter Staff, (c) Filter Location, (d) Legend of Markers, (e) map, and (f) left menu bar (Figure 4). All these areas were marked in the eye-tracking analyzing software to calculate how much time participants spent looking at these particular areas.
Figure 4. Area of interests (AOI) on the index page.
The results showed that participants spent 3 times longer (265.38 mean seconds) looking at the interactive map (see Table 4) as compared to the rest of the index page (a total of 88.13 seconds on average). Thus, the interactive map is the main tool that participants interacted with to accomplish the tasks.
Table 4. Total Visit Duration on Index Page by AOI
AOI |
Mean (seconds) |
Map |
265.38 |
Left menu bar |
46.92 |
Legend of markers |
27.79 |
Filter patient |
7.17 |
Search location |
3.11 |
Filter staff |
3.14 |
Browsing pattern on index page
The eye-tracking system also calculated the time to participants’ first fixation on each AOI. The data helped the researchers to better understand when participants looked and where they looked—sequentially. The data also helped to identify users’ browsing behavior on a given page, which reflects the importance of the various sections on the page. Users first fixated on the interactive map, then the legend of markers section in order to identify the meaning of the symbols on the map (Table 5).
Table 5. Time to First Fixation on AOIs on Index Page
Area of interest (AOI) |
Mean (seconds) |
Map |
0.69 |
Legend of markers |
4.96 |
Left menu bar |
41.92 |
Filter patient |
56.41 |
Search location |
62.96 |
Filter staff |
102.00 |
Figure 5 illustrates a participant’s reading pattern on the index page. The results show that users first fixated on the map on the screen, second, eye gazes focused on the legend of markers, and third, they sought the left menu bar in order to search and accomplish the tasks. Users then visited the filter patient section, the search location section, and gradually, the filter staff section.
Figure 5. Reading pattern on index page.
Heat maps
Heat maps provide a visualization of the fixation duration so that researchers can understand where users are fixated. Eye gaze fixations are visualized in a range of green, yellow, and red. Red sections represented the longest fixation duration, while green represented a shorter duration time. The Panacea’s Cloud™ heat map index page, a rich feature page, was created to identify where participants would focus (Figure 6). The heat map showed that participants spent the longest time fixating on the interactive map. Second, the most attractive part of the index page for participants was the legend of markers section followed by the left menu bar. The heat map is just the visual representation of the fixation duration times and helps readers to visually understand fixation duration variations on the page.
Figure 6. Heat map for the Panacea’s Cloud™ index page.
Think Aloud Interview Results
We analyzed the qualitative data obtained through participants’ comments while they were accomplishing the tasks. Participants’ think-aloud comments were reported according to each task and will be considered in future system development efforts. The index page received positive feedback to the point that users wanted to do everything on the map. Several participants recommended a live stream tab be made available on the index page in order to help the user to quickly view the incidents depicted on the map. There was a minor opinion that the system should be used more as a preparation tool than a way to assist staff in the field. One participant stated that the video is less useful in populated areas because first responder staff are well trained and establishing a connection would take time that could be used to stabilize patients.
The interviewees mainly struggled with identifying the symbols on the interactive map located on the index page. They found the search box was too small and had difficulties with sorting patients by condition. With regard to the subpage called Facilities, the interviewees’ main concern was that the facilities name did not necessarily match what was on the page. A name change to Resources was recommended by two participants. Most interviewees also wanted additional information on the facilities page.
The interview results with regard to the Video-Feeds feature showed three main concerns:
- There was not enough space between the actions. It was recommended the researchers consider adding space between User, View Only, and Call User.
- They said clicking the word, Call, was not intuitive.
- There was no place to take notes within the video feed pop-up window. They indicated that adding a note-taking feature or placing a link onto the Add Patients page directly within the video feed would be useful.
Additionally, some of the interviewees complained about overlapping icons on the interactive map view and suggested making the icons more visually appealing, rather than just different colors. Sorting patients by status was another feature that participants pointed to as important and they suggested adding a patients’ section with its own submenu.
User Satisfaction Survey Results
Adopting Sauro and Lewis’ (2012) curved grading scale (also provided in Lah & Lewis, 2016), our data can be interpreted as follows: Panacea’s Cloud™ SUS score averaged 65. A typical target for an above average user experience is a mean of 80. To this end, a curved grading scale of Panacea’s Cloud™ indicated that participants’ satisfaction was at a C level and there was room for improvement. The SUS score itself did not indicate why users were satisfied or not with the system. Therefore, we also asked users to share their thoughts about possible reasons related to user satisfaction.
Interview Results
The survey used was conducted to gain a deeper understanding of participants’ thoughts, needs, and expectations regarding the Panacea’s Cloud™ system. The results showed that current communication opportunities through existing systems (other than Panacea’s Cloud™) are not adequate for mass casualty triage according to 7 participants (n = 9). With Panacea’s Cloud™, most participants were satisfied (8 out of 9) with the audio and video quality. All participants agreed that real-time location mapping of patients and staff was crucial to achieving the goal of the tasks and increased the users’ ability to manage challenging situations. Almost all participants (9 out of 10) thought that the addition of a real-time triage beacon tracking would increase their efficiency. Most participants (8 out of 10) stated that direct real-time communication with the paramedic could save time and money, and Panacea’s Cloud™ dashboard was easy to use according to 7 participants out of 10. In addition to the mass casualty situations, half of the participants (5 out of 10) thought that they could use Panacea’s Cloud™ system in their daily routines. Some users (3 out of 10) mentioned that the system would be good during disasters but was not appropriate for the daily routine operations. They recommended redesigning the system to support operations such as intercommunications among incident commanders and the monitoring of hospital resources.
Lessons Learned
The results of our research showed that the Panacea’s Cloud™ system still needs improvement. In particular, the dashboard menus and submenus should be redesigned logically to help the users quickly browse through the site. There are some user demand features such as placing a video call button, creating the buttons for controlling video feed, and easy pinning of patients and paramedics on the interactive map with intuitive symbols and colors. The features that users demanded would impact user satisfaction and ease-of-use of the system.
Panacea’s Cloud™ would be an effective MCI management tool because 90% of the participants successfully completed the tasks, which are derived from the major tasks needed during multiple casualty events. Additionally, all tasks were completed by participants within 24 minutes and 3 seconds on average (see Table 2 for efficiency results) including familiarizing themselves with the system, checking incidents, finding paramedics at the given address, establishing connections, logging in patient information, changing patient status, and reporting. Although there is no indicator for now whether the average time taken for all this is acceptable, the next stages of development of the Panacea’s Cloud™ system will include more comparisons of the processes involved.
The eye-tracking research results revealed that incident commanders spent most of their time checking the index page of the system that hosts the interactive map for tracking the patients and paramedics. They also spent time mostly on the video feeds page when video conferencing with paramedics to help triage the patients and to gather the patients’ information. Participants want to have rapid access to all major tasks on the index page in order to accomplish their tasks as quickly and as easily as possible. Dashboard users of Panacea’s Cloud™ are able to track the patients on the map, but they need to visit another page to log them into the system or to change the patient conditions, which results in an increasing cognitive workload and a decrease in the efficiency rates of the system.
Participants were spending more time on the interactive map than any other sections of the index page. Because the dynamic map reflects all rolling activities on the map, users mostly kept their eyes on the map to track the dynamic environment of the casualty incident. Heat maps also proved that users were fixated on the map while they were accomplishing the assigned tasks. Participants’ second longest visit occurred on the left menu bar. If the major tasks were offered within the dynamic map view, users would save time on task completion according to respondents.
The time to first fixation data revealed that initially users paid attention to the map on the index page, as expected, then they moved their eyes to the legend of markers section to identify the different symbols. The symbols on the interactive map that represent the paramedics and patients with different triage events should be redesigned for providing a more meaningful way for users to easily differentiate the symbols and colors on the map.
According to the think aloud results, ICs desire to communicate to the first responders on the scene, to diagnose the conditions of the patients, to log the patient data, and to track the patients’ locations. ICs with adequate situational awareness sensors are more likely to help users render better decisions concerning the efficient use of resources, such as arranging vehicles to transfer patients to hospitals in the area, monitoring available beds and blood bags, and so on.
Considering the current user satisfaction level, the product appears close to what is expected by the industry. Redesigning Panacea’s Cloud™ with user-friendly design principles would further impact overall user satisfaction of the system.
The post-task survey results showed that current communication methods in existing systems are not adequate for mass casualty triage events. Participants commented that using a real‑time triage beacon tracking would increase efficiency. However, technical improvements were indicated.
Panacea’s Cloud™ offers a hands-free communication device that applies emerging technologies and promising solutions for optimal triaging and logging patient information when compared to existing MCI management systems. However, only half of the participants thought Panacea’s Cloud™ system—as currently designed—would be appropriate for routine daily activities. The respondents suggested that developers should consider adding new features to Panacea’s Cloud™ to foster the acceptance of the application for daily use, if that is the main goal.
Competitive Analysis with Existing System
Intermedix WebOEC™ is the existing information communication system for mass casualty incident management (http://sema.dps.mo.gov/programs/webeoc/). It does not, however, provide real-time paramedic geographical location information and does not support audio/video communication between the physician and the paramedic.
The comparison between Intermedix WebEOC™ and Panacea’s Cloud™ shown in Table 6 revealed that Panacea’s Cloud™ covers all features currently offered by the Intermedix WebEOC™ system.
Panacea’s benefit is that it can support audio/video communication, record patient information, provide the ability to change a patient’s condition, and sort patients by credentials. Additionally, recording patient information is built in to the system and tracking patients occurs in real time on an interactive map and can be sorted by name, location, incident, and status.
Table 6. Comparative Review: Panacea’s Cloud™ Versus the Intermedix WebOEC™ System
Features |
Panacea’s Cloud™ |
Intermedix WebOEC™ |
Interactive map |
Y |
Y |
List incidents |
Y |
Y |
Real-time paramedic geo info |
Y |
N |
Paramedics phone numbers |
Y |
Y |
Paramedics email addresses |
Y |
Y |
List of physicians in charge |
Y |
Y |
Audio/video communication with paramedic |
Y |
N |
Record patient info into the system |
Y |
N |
Track patient geo info |
Y |
N |
Change patient condition |
Y |
N |
Sort patients |
Y |
N |
Design Principles for AR Platforms for Disaster Management in General
Based on the empirical data shown previously, we derived the following general design recommendations for future augmented reality (AR) based MCI platforms:
Situational awareness. As integral leaders of the Federal Emergency Management Agency’s (FEMA) incident command system (https://www.fema.gov/incident-command-system-resources), a main requirement is that ICs coordinate all levels of disaster responses including operations, planning, logistics, and finance. A robust communication system is critical to the success of this coordination for any MCI. While the triage assessment is made by each first responder team locally, the task of global MCI triage is the IC’s responsibility. To accomplish this task successfully, complete situational awareness may include the most critical triage levels—availability of transport for those critical patients and the accounting of medical resources to care for them. This could be achieved by smart glasses where ICs can actually see what the paramedics see while at the scene. The next generation of AR platforms for MCI IC dashboard should provide this information in the most user-friendly and efficient manner to assist ICs in their tasks and not to hinder them. Panacea’s Cloud™ is a prototype dashboard that hopes to fulfil this role. When ICs decide to transport patients to a hospital in the area, the ability to monitor the hospital resources (i.e., blood bags, beds, available surgeons, etc.) is crucial. Situational awareness also includes a dynamic dataset that represents the hospital resources and should be included in the next iteration of this system to better help incident commanders to make optimal decisions.
Synchronous map view. Functionality and interface design is important for achieving managers’ goals at the time of disaster. In this context, incident commanders who work under stress during MCIs mostly welcomed the synchronous map view to help them accomplish their tasks as this hosts a real-time geo positioning feature for both paramedics and patients. Therefore, the next generation of AR platforms for MCI should provide the ability to interact using a map that hosts real-time information of the incident sites. Additionally, the current status of the incidents should be made visible so as to inform dashboard users about the interactive map view. Keeping all major tasks in the same place with a synchronous map is the main desire of the incident commanders. The next generation of AR platforms for MCIs should also have an IC dashboard to help track paramedics real-time geo location, call paramedics, enter patients’ information, change patients’ information, and provide a digital note taking feature. The design should be informed by a symbol based design to represent patient conditions, availability of paramedics, and the real-time movement of the incident scene. Additionally, a video calling feature, video control buttons, and displaying a video feed with a digital note taker should also be features included in the revised design.
Hands-free communication with AR smart glasses. Paramedics and ICs desire to communicate particularly during the patients’ triaging process and when trying to identify priorities with patients’ transportation. Launching an audio/visual communication system using smart glasses differentiates the next generation systems from current existing systems, which increase the capability of correct triaging, logging patient information, and tracking and transporting patients. Furthermore, incident commanders should be able to see what paramedics are looking at, which should help increase their situational awareness and decision making ability. Hence, the next generation of AR platforms for MCIs needs to include an audio and visual communication linkage between both parties. Moreover, hands-free communication is key for paramedics because paramedics should use both hands when helping victims at the time of communicating with incident commanders.
Digital notes. The next generation of AR platforms for MCI should support the ability of the provider to enter the patient’s information into the system during any MCI. Our research shows that the incident commanders presently surveyed found this feature very impressive and beneficial. An ability to record digital notes during the call was highly recommended, as well. The ICs also emphasized the need to log the patient’s information using data forms and video feeds on the same screen. Therefore, the next generation of AR platforms for MCIs should enable users to record digital notes and permit them to log the information accordingly during any communication with paramedics.
Own Wi-Fi. Our research points to the importance of a next generation MCI system that integrates alternative networks. MCI management systems such as Panacea’s Cloud™ also need their own robust and resilient Wi-Fi network that employs self-sustaining battery power. This will provide for a vital communication infrastructure on the ground where existing communication systems may have been damaged or eliminated. Additions to the network can connect the local system to the Internet grid thereby providing a link to remote ICs anywhere in the world.
Conclusion
A resilient information and communication system is needed to manage mass casualty incidents quickly and effectively during MCIs. Panacea’s Cloud™ is a recently developed Incident Management System that aims to provide an effective communication and coordination system between medical incident commanders (ICs) using a website structured dashboard and first medical responders (paramedics) at the site who wear smart glasses that support hands-free triage and communication approaches.
The usability testing of the system revealed that audio/visual communications between ICs and paramedics help in managing the incidents and in correctly triaging the patients. A dashboard with a synchronous map that hosts real-time tracking data of patients and paramedics provides visual cues for dashboard users and thus fosters ease-of-management of the incident. It is also understood from the research that ICs don’t want to browse back and forth through the website to accomplish the tasks, thus main functions for all major tasks should be hosted on the main dashboard page so as to allow users fast browsing of datasets. Additionally, appropriate terminology should be used consistently throughout the system. Icon-based simplistic designs help dashboard users capture the circumstances easily by reducing the cognitive workload. User-centered design approaches have a great impact on the acceptability of such complex interactive services, thus refinements for the next design cycle with user-centered design principles are needed to provide best tasks achievement aligned with user satisfaction. The performance of the network system might be affected by many factors particularly during MCIs, consequently it is important to consider creating a reliable Wi-Fi network for system to function properly.
Next generation MCI systems should also provide a system for appropriate triage, monitoring of patients and paramedics on-scene, that does not overwhelm hospitals and provides for effective use of resources.
Limitations and Future Research
In this study, only the dashboard used by the IC was the subject of the usability test; the smart glasses and the role of the paramedic was facilitated by a researcher. The fact that the paramedics’ had the ability to use the smart glasses was not part of the study. The paramedics role and use of smart glasses should be tested in future research.
Tips for Usability Practitioners
The following ideas were garnered from our study and would be beneficial to practitioners who design user interfaces for multiple casualty incidents (MCI):
- There is a need to rethink and develop a new effective and efficient MCI management system with immersive and augmented reality applications.
- An expert review of any prototype would help researchers to identify design and labeling errors early on before conducting usability testing.
- Tasks tested should be derived from real situations at the time of disaster. In this way participants might contribute in product development with high-quality ideas.
- A synchronous map with intuitive symbols would increase the efficiency of users in charge of rescue operations.
- Keeping all major tasks in the dashboard area allows users to carry out fast browsing of data sets with less effort.
- Usability studies usually focus on users’ satisfaction; however, a new method is required that tests the alignment of users’ satisfaction and successful task achievements (especially for MCIs where lives are often in danger).
Acknowledgements
This material is based upon work supported by the Wallace H. Coulter Foundation, Translational Partnership Program (http://whcf.org/). We are thankful for the contributions by So Mi Kim, postdoctoral fellow at the School of Information Science and Learning Technologies; Olivia Apperson, Dmitrii Chemodanov, Kendall Park, John Gillis, Graduate Student Assistants at the Department of Computer Science; Kenneth Haggerty, Kodjo Atiso, Hao He, Graduate Student Assistants at the School of Information Science and Learning Technologies; and Neeley Current, Manager of the Information Experience Lab at the School of Information Science and Learning Technologies.
References
Aguiar, Y. P. C., Vieira, M. de F. Q., Galy-Marie, E., & Santoni, C. (2015). Analysis of the user behaviour when interacting with systems during critical situations. In J.-M. Mercantini & C. Faucher (Eds.), Risk and Cognition (pp. 129–154). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-45704-7_6
Artinger, E., Maier, P., Coskun, T., Nestler, S., Mähler, M., Yildirim-Krannig, Y., … Klinker, G. (2012). Creating a common operation picture in realtime with user-centered interfaces for mass casualty incidents. In 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops (pp. 291–296). https://doi.org/10.4108/icst.pervasivehealth.2012.248598
Aylwin, C. J., König, T. C., Brennan, N. W., Shirley, P. J., Davies, G., Walsh, M. S., & Brohi, K. (2006). Reduction in critical mortality in urban mass casualty incidents: Analysis of triage, surge, and resource use after the London bombings on July 7, 2005. The Lancet, 368(9554), 2219–2225. https://doi.org/10.1016/S0140-6736(06)69896-6
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–94.
Berndt, H., Mentler, T., & Herczeg, M. (2016). Smartglasses for the triage of casualties and the identification of hazardous materials. I-Com, 15(2), 145–153. https://doi.org/10.1515/icom-2016-0024
Bishop, P. A., & Herron, R. L. (2015). Use and misuse of the Likert item responses and other ordinal measures. International Journal of Exercise Science, 8(3), 297–302.
Blomqvist, E. (2014). The use of semantic web technologies for decision support – A survey. Semant. Web, 5(3), 177–201.
Blum, J. R., Eichhorn, A., Smith, S., Sterle-Contala, M., & Cooperstock, J. R. (2013). Real-time emergency response: Improved management of real-time information during crisis situations. Journal on Multimodal User Interfaces, 8(2), 161–173. https://doi.org/10.1007/s12193-013-0139-7
Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. In IEEE Transactions on Professional Communication, 43(3), pp. 261–278.
Danielsson, M., & Alm, H. (2012). Usability and decision support systems in emergency management. Work (Reading, Mass.), 41 Suppl 1, 3455–3458. https://doi.org/10.3233/WOR-2012-0624-3455
Demir, F. (2015). Crisis communication management within the context of strategic communication and perception management Stratejik iletişim ve algı yönetimi bağlamında kriz iletişim yönetimi. Journal of Human Sciences, 12(1), 343–362. https://doi.org/10.14687/ijhs.v12i1.3161
Demir, F., Karakaya, M., & Tosun, H. (2012). Research methods in usability and interaction design : Evaluations and case studies (2nd ed.). Germany: LAP LAMBERT Academic Publishing.
Doeweling, S., Tahiri, T., Sowinski, P., Schmidt, B., & Khalilbeigi, M. (2013). Support for collaborative situation analysis and planning in crisis management teams using interactive tabletops. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (pp. 273–282). New York, NY, USA: ACM. https://doi.org/10.1145/2512349.2512823
Ellebrecht, N., Feldmeier, K., & Kaufmann, S. (2013). IT’s about more than speed. The impact of IT on the management of mass casualty incidents in Germany. In F. F. T. Comes & 10th International ISCRAM Conference on Information Systems for Crisis Response and Management (Eds.), ISCRAM 2013 Conference Proceedings – 10th International Conference on Information Systems for Crisis Response and Management (pp. 391–400). KIT; Baden-Baden: Karlsruher Institut fur Technologie. Retrieved from http://idl.iscram.org/files/ellebrecht/2013/473_Ellebrecht_etal2013.pdf
EMS, EMTs & Paramedics – Prehospital Emergency Medicine | EMS1. (n.d.). Retrieved September 23, 2016, from http://www.ems1.com/
Ferguson, H. T., Gesing, S., & Nabrzyski, J. (2016). Measuring usability in decision tools supporting collaborations for environmental disaster response. In 2016 49th Hawaii International Conference on System Sciences (HICSS; pp. 2872–2881). https://doi.org/10.1109/HICSS.2016.360
Ganz, A., Schafer, J. M., Yang, Z., Yi, J., Lord, G., & Ciottone, G. (2014). Mobile DIORAMA-II: Infrastructure less information collection system for mass casualty incidents. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 2682–2685). https://doi.org/10.1109/EMBC.2014.6944175
Gillis, J., Calyam, P., Bartels, A., Popescu, M., Barnes, S., Doty, J., … Ahmad, S. (2015). Panacea’s glass: Mobile cloud framework for communication in mass casualty disaster triage. In 2015 3rd IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud; pp. 128–134). https://doi.org/10.1109/MobileCloud.2015.39
Jiang, D., Huang, R., Calyam, P., Gillis, J., Apperson, O., Chemodanov, D., … Salman, A. (2016). Hierarchical cloud-fog platform for communication in disaster incident coordination (Under Review).
Kattimani, V. S., Tiwari, R. V., Pandi, S. C., Meka, S., & Lingamaneni, K. P. (2015). Disaster management and the role of oral maxillofacial surgeons. Journal of Clinical & Diagnostic Research, 9(12), 1–4. https://doi.org/10.7860/JCDR/2015/14436.6892
Koning, S. W., Ellerbroek, P. M., & Leenen, L. P. H. (2014). Indoor fire in a nursing home: Evaluation of the medical response to a mass casualty incident based on a standardized protocol. European Journal of Trauma and Emergency Surgery, 41(2), 167–178. https://doi.org/10.1007/s00068-014-0446-z
Lah, U., & Lewis, J. R. (2016). How expertise affects a digital-rights-management-sharing application’s usability. IEEE Software, 33(3), 76–82. https://doi.org/10.1109/MS.2015.104
Lee, W. B., Wang, Y., Wang, W. M., & Cheung, C. F. (2012). An unstructured information management system (UIMS) for emergency management. Expert Systems with Applications, 39(17), 12743–12758. https://doi.org/10.1016/j.eswa.2012.02.037
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale (PDF). International conference (HCII 2009), San Diego CA, USA.
Ley, B., Ludwig, T., Pipek, V., Randall, D., Reuter, C., & Wiedenhoefer, T. (2014). Information and expertise sharing in inter-organizational crisis management. Computer Supported Cooperative Work (CSCW), 23(4–6), 347–387. https://doi.org/10.1007/s10606-014-9205-2
Liao, T. (2016). Is it ‘augmented reality’? Contesting boundary work over the definitions and organizing visions for an emerging technology across field-configuring events. Information and Organization, 26(3). https://doi.org/10.1016/j.infoandorg.2016.05.001
Lixin, Y., Lingling, G., Dong, Z., Junxue, Z., & Zhanwu, G. (2011). An analysis on disasters management system in China. Natural Hazards, 60(2), 295–309. https://doi.org/10.1007/s11069-011-0011-6
May, A., Mitchell, V., & Piper, J. (2014). A user centred design evaluation of the potential benefits of advanced wireless sensor networks for fire-in-tunnel emergency response. Fire Safety Journal, 63, 79–88. https://doi.org/10.1016/j.firesaf.2013.11.007
Meissner, A., Luckenbach, T., Risse, T., Kirste, T., & Kirchner, H. (2002). Design challenges for an integrated disaster management communication and information system. In DIREN 2002 – 1st IEEE Workshop on Disaster Recovery Networks; New York, 2002. Retrieved from http://www.l3s.de/ risse/pub/P2002-01.pdf
Mentler, T., & Herczeg, M. (2015). Interactive cognitive artifacts for enhancing situation awareness of incident commanders in mass casualty incidents. Journal of Interaction Science, 3(1). https://doi.org/10.1186/s40166-015-0012-0
Nielsen, J. (1992). The usability engineering life cycle. Computer, 25(3), 12–22. https://doi.org/10.1109/2.121503
Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods. John Wiley & Sons.
Nielsen, J., Clemmensen, T., & Yssing, C. (2002). Getting access to what goes on in people’s heads?: Reflections on the think-aloud technique. In Proceedings of the Second Nordic Conference on Human-computer Interaction (pp. 101–110). New York, NY, USA: ACM. https://doi.org/10.1145/572020.572033
Paelke, V., Nebe, K., Geiger, C., Klompmaker, F., & Fischer, H. (2012). Designing multimodal map-based interfaces for disaster management. In Proc ACHI 2012.
Park, J., Cullen, R., & Smith-Jackson, T. (2014). Designing a decision support system for disaster management and recovery. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 1993–1997. https://doi.org/10.1177/1541931214581416
Reddy, M. C., Paul, S. A., Abraham, J., McNeese, M., DeFlitch, C., & Yen, J. (2009). Challenges to effective crisis management: Using information and communication technologies to coordinate emergency medical services and emergency department teams. International Journal of Medical Informatics, 78(4), 259–269. https://doi.org/10.1016/j.ijmedinf.2008.08.003
Russo, R. M., Galante, J. M., Jacoby, R. C., & Shatz, D. V. (2015). Mass casualty disasters: Who should run the show? Journal of Emergency Medicine, 48, 685–692. https://doi.org/10.1016/j.jemermed.2014.12.069
Salman, Y. B., Cheng, H.-I., & Patterson, P. E. (2012). Icon and user interface design for emergency medical information systems: A case study. International Journal of Medical Informatics, 81(1), 29–35. https://doi.org/10.1016/j.ijmedinf.2011.08.005
Sauro, J. (2011a). Measuring usability with the System Usability Scale (SUS): MeasuringU. Retrieved May 11, 2016, from http://www.measuringu.com/sus.php
Sauro, J. (2011b). SUStisfied? Little-known System Usability Scale facts. User Experience Magazine. Retrieved May 11, 2016, from http://uxpamagazine.org/sustified/
Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Waltham, MA: Morgan Kaufmann.
Vassell, M., Apperson, O., Calyam, P., Gillis, J., & Ahmad, S. (2016). Intelligent dashboard for augmented reality based incident command response co-ordination. In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC; pp. 976–979). IEEE. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7444921
WEBEOC. (2011). Retrieved October 11, 2016, from http://www.chathamemergency.org/eoc/webeoc-training.php