Abstract
IBM was contracted to provide a new Air Defence Command and Control (ADCC) system for the Royal Air Force. The IBM Human Factors (HF) team was responsible for the design of the operations room, workstations and the graphical user interfaces. Because the project was safety-related, IBM had to produce a safety case. One aspect of the safety case was a demonstration of the operational effectiveness of the new system.
This paper is an in-depth case study of the user testing that was carried out to demonstrate the effectiveness of the system. Due to time constraints the HF team had to observe five participants working simultaneously. Further, to provide a realistic operational environment, up to twenty-eight operators were required for each test. The total effort for this activity was four person-years. The paper will detail the considerations, challenges and lessons learned in the creation and execution of these multi-user user tests.
Practitioner’s Take Away
- Although the client may specify or mandate a UCD approach (for example, ISO 13407 [3] or Defence Standard 00-25 [2]) to be followed, they may not realise how expensive it is to support this approach. The reduction in whole life cycle costs of using a UCD approach are well documented, but the costs of supporting the development from the customer’s perspective are sometimes less explicit. Therefore, it is worthwhile discussing this with the client early in the programme, and if necessary, updating the risk register accordingly.
- The summative user tests are an excellent mechanism for collecting data and evidence for a safety case. However, to ensure that appropriate safety-related data can be collected on a safety-related system the participants must be placed in an appropriate working environment under a realistic amount of workload.
- Do not rely solely on a summative user test to produce all the necessary operational safety-related data for a safety case. A good design process that includes activities such as formative user tests and heuristic evaluations, as well as the demonstration of effective training materials can also produce an enormous amount of safety-related data [4].
- The production of the training materials must be aligned with the user tests. The Training team must be notified as early as possible on the required roles and tasks for each user test. This will ensure that the appropriate training materials are available at the right time. Further, Train The Trainer materials are not usually appropriate for a user test. They are often lacking in context and standard operating procedures as these are usually added later by the client’s Trainers once they have been trained. On this project, it took the HF team 60 days to convert the materials into the correct format. Therefore, design the training materials so they can be used effectively for both purposes (i.e. TTT and user tests).
- Limit the number of observers to only those that are absolutely necessary. When these tests were carried out, the project was nearing completion and everybody wanted to see the system in action. This was a cause of concern for the HF team as the more people that are around the more chance of distraction and skewing of results, albeit accidentally. Therefore, create and distribute observation etiquette instructions to everybody entering the user test area. Even attach copies of the instructions to the test room entrance, in the canteen and in congregation areas.
