Case Study: Conducting Large-Scale Multi-User User Tests on the United Kingdom Air Defence Command and Control System

Peer-reviewed Article

pp. 121-135Download full article (PDF)

Abstract

IBM was contracted to provide a new Air Defence Command
and Control (ADCC) system for the Royal Air Force. The IBM Human Factors
(HF) team was responsible for the design of the operations room, workstations
and the graphical user interfaces. Because the project was safety-related,
IBM had to produce a safety case. One aspect of the safety case was a demonstration
of the operational effectiveness of the new system.

This paper is an in-depth case study of the user testing
that was carried out to demonstrate the effectiveness of the system. Due
to time constraints the HF team had to observe five participants working
simultaneously. Further, to provide a realistic operational environment,
up to twenty-eight operators were required for each test. The total effort
for this activity was four person-years. The paper will detail the considerations,
challenges and lessons learned in the creation and execution of these multi-user
user tests.

Practitioner’s Take Away

  • Although the client may specify or mandate a UCD approach (for example,
    ISO 13407 [3] or Defence Standard 00-25 [2]) to be followed, they may
    not realise how expensive it is to support this approach. The reduction
    in whole life cycle costs of using a UCD approach are well documented,
    but the costs of supporting the development from the customer’s
    perspective are sometimes less explicit. Therefore, it is worthwhile discussing
    this with the client early in the programme, and if necessary, updating
    the risk register accordingly.
  • The summative user tests are an excellent mechanism for collecting data
    and evidence for a safety case. However, to ensure that appropriate safety-related
    data can be collected on a safety-related system the participants must
    be placed in an appropriate working environment under a realistic amount
    of workload.
  • Do not rely solely on a summative user test to produce all the necessary
    operational safety-related data for a safety case. A good design process
    that includes activities such as formative user tests and heuristic evaluations,
    as well as the demonstration of effective training materials can also
    produce an enormous amount of safety-related data [4].
  • The production of the training materials must be aligned with the user
    tests. The Training team must be notified as early as possible on the
    required roles and tasks for each user test. This will ensure that the
    appropriate training materials are available at the right time. Further,
    Train The Trainer materials are not usually appropriate for a user test.
    They are often lacking in context and standard operating procedures as
    these are usually added later by the client’s Trainers once they
    have been trained. On this project, it took the HF team 60 days to convert
    the materials into the correct format. Therefore, design the training
    materials so they can be used effectively for both purposes (i.e. TTT
    and user tests).
  • Limit the number of observers to only those that are absolutely necessary.
    When these tests were carried out, the project was nearing completion
    and everybody wanted to see the system in action. This was a cause of
    concern for the HF team as the more people that are around the more chance
    of distraction and skewing of results, albeit accidentally. Therefore,
    create and distribute observation etiquette instructions to everybody
    entering the user test area. Even attach copies of the instructions to
    the test room entrance, in the canteen and in congregation areas.