UX Standards and UX Maturity

Abstract

The International Standard ISO 9241-11 is frequently referenced as a source definition for Usability. In the past 10 years, a number of standards that relate to Usability have been published. In this essay, I highlight the ISO standards that focus on human-centered design and quality information processes, and I identify the technical committees responsible for the development of these standards. The ISO process capability model and framework for assessment provides a structure for examination of organizational UX Maturity. As our UX processes become embedded in product life cycle development, UX standards may increase in importance in areas such as certification, regulatory approval, and interoperability. Some barriers continue to deter the widespread use of ISO standards, and these include cost and complex language.

Keywords

ISO, UX Maturity, UX process capability, Quality

Introduction

Highly regulated industries such as Telecommunications, Aerospace, and Finance are expected to demonstrate conformance to standard processes or regulatory specifications. In many cases, this demonstration is done using independent organizations such as accredited certification bodies. A demonstration of quality capability can be a requirement for certain services and contracts. It can also be used to ensure interdependencies—between services, products, suppliers, vendors, and customers—meet standards.

In the first issue of the Journal of Usability Studies, Mary Theofanos and Whitney Quesenbery presented a paper on best practices for reports of formative usability evaluations (2005). The practical takeaway indicated that there was little guidance available for these types of reports, and there was significant variation in reporting. Since 1998, Rolf Molich has championed 10 Comparative Usability Evaluation (CUE) studies to examine reproducibility in usability evaluations with practicing professionals and academics (n.d.). A consistent theme through the CUE studies has been the variability in practice and reporting (Molich, 2018). While experienced practitioners are able to identify usability issues and make corresponding recommendations for solutions, the variability in findings and how reporting is carried out suggests the need for stronger guidance.

The May 2019, Journal of Usability Studies was dedicated to the work done by Nigel Bevan. The invited essay by Tom Tullis (2019) provided some historical context to the early work on UX standards championed by Nigel and many others. Bevan (2009) identified the barriers to standards use and highlighted four subject areas:

• User Interface Design
• Usability Assurance
• Usability and Software Quality
• Human-Centered Design Process

Consider the last two items as co-dependent, where guidance on usability and quality measures can be tied to design processes. In this essay, I would like to highlight some relevant standards and then link how these can support UX maturity and UX certification.

Standards are developed through consensus agreement by national and international standards organizations. Standards are not easy to navigate and have complex interdependencies. Some of the standards most relevant to UX practitioners are presented below.

The Standards Vocabulary

Perhaps the best-known Usability standard (probably the most referenced for the definition of Usability) is the ISO 9241-11:2018 Ergonomics of human-system interaction—Usability: Definitions and concepts.

You can locate this document with a query on the ISO website (iso.org). Here you can view the abstract, the TOC, and non-restricted content such as the front matter (Forward, Introduction, Scope, Terms and Definitions, and the Bibliography). The structure of ISO standards documents is consistent. You can view the scope and get an overview of content by reviewing the TOC. The definitions can be a good indicator of important terms used in the document, and the Bibliography allows you to get a sense of related and associated documents. In order to access the complete document, there is a fee (138 Swiss Francs, approximately 150 USD).

You can glean some additional information from the query results such as the Technical Committee responsible for the standard, version number, edition, and number of pages. Standards are also organized by the International Standard Catalogue (ICS), which consists of numbered groupings by area. For ISO 9241-11:2018, these are

• Technical Committee : ISO/TC 159/SC 4 Ergonomics of human-system interaction and
• ICS : 13.180 Ergonomics 35.180 IT terminal and other peripheral equipment.

In this instance, you will find that the technical committee has responsibility for standards that fall into the subject area of Ergonomics of human-system interaction. This committee has published 83 standards with another 10 under development. The scope of responsibility of this committee includes the following:

• Hardware ergonomics (input display and interactive devices, associated workplaces, and environments)
• Software ergonomics (dialogue and interaction design)
• Human-centered design processes and methods

For the purpose of this essay, I will focus on standards for human-centered design processes and usability related information.

The ISO 9241-11:2018 standard is classified under two subject areas, ICS 13 – Environmental, Health and Safety, and ICS 35 Information Technology. This allows for input and development to be coordinated.

Working Groups

The following are the two working groups of particular importance to UX practitioners:

• ISO/TC 159/SC 4/WG 6 – Human-centred design processes for interactive systems
• ISO/TC 159/SC 4/JWG 28 – Joint ISO/TC 159/SC 4 – ISO/IEC JTC 1/SC 7 WG: Common industry formats for usability related information

The focus of ISO/IEC JTC 1/SC 7 is on Software and Systems Engineering. This joint technical committee has a scope that addresses standardization of processes, supporting tools, and supporting technologies for the engineering of software products and systems. A Joint Working Group (JWG) is where members of two different subject areas work together, and this occurs in JWG 28. This working group has been responsible for a series of publications related to Common Industry Formats (CIFs) for various usability related reports and processes.

Each working group determines what topics might benefit from having new standards defined and is also responsible for periodic reviewing and updating of existing published standards. ISO has a numbering system (stages) for identifying work in progress as well as what is published. These progress from Stage 00 to Stage 60.60 for published, Stage 90 for standards under review, and Stage 95 for those no longer published.

Table 1. ISO Numbering System

Stage	Name
00	Preliminary
10	Proposal
20	Preparatory
30	Committee
40	Enquiry
50	Approval
60	Publication
90	Review
95	Withdrawal

More details on sub-stages can be found on the ISO website. Standards in the Enquiry Stage are open for comments by members of the public through national bodies. On the International Harmonized Stage Codes page, you can view the stages in a summary table and download a printable version. Subject matter experts, representing national bodies/organizations, can contribute and vote as documents are developed through a consensus review process.

Looking at the information on ISO 9241 you can learn that the current 2018 standard replaced the 1998 standard. The 60:60 Stage designation indicates that this particular version has successfully passed through the earlier development and review stages. ISO standards are examined for currency every five years. Depending on the significance of the updates, the stages 00 through 50 ensure input, collaboration, and consensus before publication.

UX Standards and UX Maturity

There are several ISO standards that focus on UX quality. These standards also tie back to standards dealing with capability assessment. A number of companies use capability assessment models to recognize process maturity within organizations. Such models can provide frameworks for growth and development of UX teams.

Standards provide guidance to development organizations and national bodies with the desire to set expectations for quality, interoperability, sustainability, and so on. New standards can be developed in parallel with technology advances and work practices. Some UX standards that focus on software quality and process frameworks are introduced in the following section.

Software Quality—SQuaRE

A series of documents dealing with usability quality can be found in the ISO/IEC 25000 series. The summary document of this series is ISO/IEC 25000:2014 Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE. The SQuaRE series consist of five numbered divisions:

• ISO/IEC 2500n – Quality Management
• ISO/IEC 2501n – Quality Model
• ISO/IEC 2502n – Quality Measurement
• ISO/IEC 2503n – Quality Requirements
• ISO/IEC 2504n – Quality Evaluation Division

There are 21 SQuaRE standards available and under development. You can find links to each standard from the ISO’s Standards by ISO/IEC JTC 1/SC7 page. From this list, you can see the Stage number and the ISO Standards Catalog (ICS) number. These numbers convey important information. For example, the ISO/IEC 25001:2014 Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—Planning and management (that was reviewed and confirmed in 2020) has a Stage number of 90.93 that indicates the appropriate review was done. The ICS number is 35:080; the 35 indicates it falls under the Information Technology category, and 080 indicates the subcategory of Software (including software documentation and use).

The following are the 21 SQuaRE standards that are under the direct responsibility of ISO/IEC JTC1 SC7, starting with the Guide to SQuaRE (note that I have shortened each standard’s name, for example, from “Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)” to “Systems and software engineering—(SQuaRE)”):

• ISO/IEC 25000:2014 Systems and software engineering—(SQuaRE)—Guide to SQuaRE
• ISO/IEC 25001:2014 Systems and software engineering—(SQuaRE)—Planning and management
• ISO/IEC CD 25002.2 Systems and software engineering—(SQuaRE)—Quality models overview and usage
• ISO/IEC 25010:2011 Systems and software engineering—(SQuaRE)—System and software quality models
• ISO/IEC CD 25010 Systems and software engineering—(SQuaRE)—Product quality model
• ISO/IEC TS 25011:2017 Information technology—(SQuaRE)—Service quality models
• ISO/IEC 25012:2008 Software engineering—(SQuaRE)—Data quality model
• ISO/IEC CD 25019.2 Systems and software engineering—(SQuaRE)—Quality in-use model
• ISO/IEC 25020:2019 Systems and software engineering—(SQuaRE)—Quality measurement framework
• ISO/IEC 25021:2012 Systems and software engineering—(SQuaRE)—Quality measure elements
• ISO/IEC 25022:2016 Systems and software engineering—(SQuaRE)—Measurement of quality in use
• ISO/IEC 25023:2016 Systems and software engineering—(SQuaRE)—Measurement of system and software product quality
• ISO/IEC 25024:2015 Systems and software engineering—(SQuaRE)—Measurement of data quality
• ISO/IEC TS 25025:2021 Information technology—(SQuaRE)—Measurement of IT service quality
• ISO/IEC 25030:2019 Systems and software engineering—(SQuaRE)—Quality requirements framework
• ISO/IEC 25040:2011 Systems and software engineering—(SQuaRE)—Evaluation process
• ISO/IEC AWI 25040 Systems and software engineering—(SQuaRE)—Managerial framework for quality evaluation
• ISO/IEC 25041:2012 Systems and software engineering—(SQuaRE)—Evaluation guide for developers, acquirers and independent evaluators
• ISO/IEC 25045:2010 Systems and software engineering—(SQuaRE)—Evaluation module for recoverability
• ISO/IEC 25051:2014 Software engineering—(SQuaRE)—Requirements for quality of Ready to Use Software Product (RUSP) and instructions for testing
• ISO/IEC DTS 25052-1.2 Systems and software engineering—(SQuaRE): cloud services—Part 1: Quality model

Human-Centered Design (HCD) Process Standards

The ISO standards 9241-220 and 9241-221 provide guidance on process examination specific to human centric design. As you can see in the standard titles, all three are grouped under Ergonomics of human and system interaction. They each have ICS identifiers (ISO Standards Catalog) that indicate the responsibility of Ergonomics (13.180) and IT and other peripheral equipment (35.180).

• ISO 9241-210:2019 Ergonomics of human-system interaction—Part 210: Human-centred design for interactive systems
• ISO 9241-220:2019 Ergonomics of human-system interaction—Part 220: Processes for enabling, executing and assessing human-centred design within organizations
• ISO/CD 9241-221 Ergonomics of human-system interaction—Part 221: HCD Process Assessment Model (PAM) and Process Reference Model (PRM)

In particular, the Process Assessment Model and the Process Reference Models provide a framework for developing organizational capability. In 9241-220, the content covers the following:

The processes are described from the viewpoint of those responsible for the analysis, design and evaluation of the human use of interactive systems. The process descriptions include the purpose, benefits, outcomes, typical activities and work products for each process, and are for use in the specification, implementation, assessment and improvement of the activities used for human-centred design and operation in any type of system life cycle. They can also provide the basis for professional development and certification (ISO Standard No. ISO 9241-220:2019, Abstract).

These standards tie back to ISO 33020:2019 Information Technology—Process assessment—Process measurement framework for assessment of process capability standard, and they are in accordance with the requirements identified in ISO/IEC 33004:2015 Information technology—Process assessment—Requirements for process reference, process assessment and maturity models.

The process assessment and maturity models provide a framework for examining quality of design and development. Table 2 presents the 5-level (6-level if considering Level 0) capability model from ISO/IEC 33020.

Table 2. The 5-Level Process Capability Model with Process Attributes Identified

Level	Process Label and [Attribute]
5	Innovating Process
	[Process innovation process attribute]
	[Process innovation implementation process attribute]
4	Predictable Process
	[Quantitative analyses process attribute]
	[Quantitative control process attribute]
3	Established Process
	[Process definition process attribute]
	[Process deployment implementation process attribute]
2	Managed Process
	[Performance management process attribute]
	[Work product management process attribute]
1	Performed Process
	[Process performance process attribute]
0	Incomplete Process purpose

Source: ISO/IEC 33020:2019 Information technology—Process assessment—Process measurement framework for assessment of process capability.

Process Performance Indicators refer to Level 1; Process Capability Indicators refer to Level 1 and above. The ISO/IEC 9241-221 standard is currently under development and considerable detail is provided on process purpose, process benefit, process outcomes, best practices, input work products, and output work products. Detailed appendixes highlight examples and labeling conventions. Table 3 presents the recommended ISO 33020 rating scale.

Table 3. ISO 33020 Rating Scale

Definition	Scale
Not achieved	0 – ≤ 15%
Partially achieved	15% – ≤ 50%
Largely achieved	50% ≤ 85%
Fully achieved	85% ≤ 100%

This collection of ISO/IEC 250xx SQuaRE standards and the 9241-210, -220 and -221 Process standards provide a pathway for the examination of UX quality in an organization.

UX Maturity—Many Models in Play

The intersection of standards and quality can be captured with a maturity-capability model. Examining your team or organization processes through the lens of a maturity model gives you insights into areas for improvement and helps you identify process strengths that have evolved over time. A number of UX Maturity Models are promoted by industry consultants and even internally within larger organizations.

For example, the NN/Group and Dialog Design both use a 6-stage maturity model. The labels in such models are in context for a UX organization.

Table 4. 6-Stage Maturity Model

Stage	Name (NN/Group)	Name (Dialog Design)
6	User-driven	Innovating
5	Integrated	Predictable
4	Structured	Established
3	Emergent	Managed
2	Limited	Performed
1	Absent	Incomplete

The NN/Group (Pernice, 2021) describes organizational behaviors that are observed at each stage. While the Dialog Design (n.d.) maturity model is derived from the ISO 33020 Standard. Descriptive UX characteristics are identified for each level such that performance indicators can then be used for compliance purposes. A competency scale, using UX language/vocabulary, results in a well-structured assessment framework. Such approaches parallel what is covered in ISO/IEC 33020 and ISO 9241-221. These are just two examples where UX maturity models are promoted for examining organizational capability and consistency.

Links to Certification

The relationship between standards and compliance assessment is prevalent in many industries. For example, international manufacturing companies proudly fly ISO 9000 flags outside of their facilities. As organizations promote and offer various forms of UX Certification, it is important that students and practicing professionals are made aware of current standards. If we are promoting process-driven activities that result in consistent levels of quality, then ISO standards provide roadmaps for expectations and assessments. An example where the interdependence between ISO Standards and the development process is explicitly documented is in the public materials made available by the International Usability and UX Qualification Board (UXQB) where the evaluation of designs against user requirements are continuously reported as part of the development process.

Figure 1. Flow for human-centered design for the ISO Standard No. 9241-210. Adapted from CPUX-F Curriculum and Glossary – Version 3.6. (https://uxqb.org/public/documents/CPUX-F_EN_Curriculum-and-Glossary.pdf). Copyright 2020 The User Experience Qualification Board, www.uxqb.org.

U.S. Food & Drug Administration Guidelines around Testing and Quality Systems

As you might expect, there are strict expectations for demonstrating adherence to design processes when it comes to medical devices and software systems. Although I haven’t worked in this field, the U.S. Food & Drug Administration (FDA) has a strict code that is outlined on their website. The following is an excerpt that addresses medical devices and a quality audit.

Each manufacturer shall establish procedures for quality audits and conduct such audits to assure that the quality system is in compliance with the established quality system requirements and to determine the effectiveness of the quality system. Quality audits shall be conducted by individuals who do not have direct responsibility for the matters being audited. Corrective action(s), including a reaudit of deficient matters, shall be taken when necessary. A report of the results of each quality audit, and reaudit(s) where taken, shall be made and such reports shall be reviewed by management having responsibility for the matters audited. The dates and results of quality audits and reaudits shall be documented (U.S. Food & Drug Administration, n.d., 21CFR820.22).

Section 820.30, Design controls, of the same code from the FDA provides guidance for devices automated with computer software to include the following:

• Design and development planning
• Design input
• Design output
• Design review
• Design verification
• Design validation
• Design transfer
• Design changes
• Design history file

Conclusion

The need for process-driven design systems with evaluation and reporting requires effort and resources. The variety of usability test methods and complexity of product life cycle development contribute to variability in the quality and repeatability of UX outcomes. For organizations with a strong focus on process, the ISO standards in the 25000 SQuaRE series and the 9241-210, -220, -221 provide guidance on quality measures. These standards have been developed to support industry and national bodies. As our UX products become embedded in complex systems, these standards can take on increasing importance and support organizations in assessing their process capability and maturity.

A deterrent to the widespread use of standards is the cost associated with accessing ISO documents. This is compounded by the fact that standards documents are difficult to read and interpret. Unless you are involved in a highly regulated industry, you may not be aware of these standards. The ISO 9241-11 is often mentioned as the source for Usability definitions, but I think few practitioners are aware of the many related standards now in place.

I don’t believe much attention has been given to UX standards in university curricula. This could be rectified by lobbying national standards bodies for free student access to such resources. University libraries could obtain appropriate access if these standards are shown to be actively used. In fact, many universities provide access to ISO/IEEE/IEC standards in Systems and Software Engineering through the IEEE Xplore database (IEEE Xplore). And of course, the World Wide Web Consortium (W3C) provides open access to its standards.

Tips for Usability Practitioners

Standards documentation are challenging to read and understand. Their structure and formal language are in part the result of format constraints and consensus agreement. For our UX community, the development of these standards signals maturity in processes, methods, and organizational positions. There is still much work to be done such that repeated levels of quality can be achieved and measured across industry sectors and organizations.

Consider getting involved in the development of standards. Most countries have an established standards body, and they are constantly seeking Industry and Academic experts to engage in review and development work. There are also professional organizations with direct involvement in standards, for example, the Institute of Electrical and Electronics Engineers. The World Wide Web Consortium has Web and Accessibility standards that are also good places to contribute.

Acknowledgments

I would like to express my sincere thanks to the editors of JUS for inviting me to contribute to this issue. Their support and guidance on the initial draft allowed me to develop and refine the content covered in this article.

References

Bevan, N. (2009). International standards for usability should be more widely used. Journal of Usability Studies, 4(3), 106–113.

International Organization for Standardization. (2014). Systems and software Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE (ISO Standard No. ISO/IEC 25000:2014). https://www.iso.org/standard/64764.html

International Organization for Standardization. (2015). Process assessment—Requirements for process reference, process assessment and maturity models (ISO Standard No. ISO/IEC 33004:2015). https://www.iso.org/standard/54178.html

International Organization for Standardization. (2018). Ergonomics of human-system interaction Part 11: Usability: Definitions and concepts (ISO Standard No. ISO 9241-11:2018). https://www.iso.org/standard/63500.html#:~:text=ISO%209241%2D11%3A2018%20provides,services%20(including%20technical%20and%20personal

International Organization for Standardization. (2019). Ergonomics of human-system interaction—Part 210: Human-centred design for interactive systems. (ISO Standard No. ISO 9241-210:2019). https://www.iso.org/standard/77520.html

International Organization for Standardization. (2019). Ergonomics of human-system interaction — Part 220: Processes for enabling, executing and assessing human-centred design within organizations. (ISO Standard No. ISO 9241-220:2019). https://www.iso.org/standard/63462.html

International Organization for Standardization. (2019). Information technology—Process assessment — Process measurement framework for assessment of process capability. (ISO Standard No. ISO 33020:2019). https://www.iso.org/standard/78526.html

International Organization for Standardization. (under development). Ergonomics of human-system interaction—Part 221: HCD Process Assessment Model (PAM) and Process Reference Model (PRM). (ISO Standard No. ISO/CD 9241-221). https://www.iso.org/standard/81384.html

Molich, R. (n.d.). CUE-studies. Dialog Design. https://www.dialogdesign.dk/cue-studies/

Molich, R. (n.d.). UX maturity—for UX professionals. Dialog Design. https://www.dialogdesign.dk/a-ux-maturity-scale/.

Molich, R. (2018). Are usability evaluations reproducible? Interactions, 25(6), 82–85. https://dl.acm.org/doi/10.1145/3278154

Pernice, K., Gibbons, S., Moran, K., & Whitenton, K. (2021). The 6 levels of UX maturity. NN/g Nielsen Norman Group. https://www.nngroup.com/articles/ux-maturity-model/

Theofanos, M., & Quesenbery, W. (2005). Towards the design of effective formative test reports. Journal of Usability Studies, 1(1), 28–46.

Tullis, T. (2019) Nigel Bevan: An overview of his contributions to usability and UX. Journal of Usability Studies, 14(3), 134–144.

U.S. Food & Drug Administration (n.d.). Code of Federal Regulations, Title 21, Volume 8, 21CFR820.22. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=820.22

UXQB (2020). International Usability and User Experience Qualification Board. CPUX-F: Curriculum and glossary (Version 3.16). https://uxqb.org/en/documents/

Published in: in Volume 17, Issue 2,