Abstract
Procuring a complex IT system is a challenge for public and private organizations regardless of whether they obtain a ready-made or customized solution. Challenges include proper definitions of requirements, technical implementation, and organizational integration. Because many user groups within an organization use these complex IT systems for different tasks, usability is multifaceted. This study explored the relationship between complexity and usability while procuring IT systems. Our research consisted of two parts: We conducted a case study at a Finnish public organization, including interviews with seven IT professionals with extensive experience in the procurement and implementation of complex IT systems. In a scoping review, we identified 106 articles, of which 28 met the inclusion criteria and were included in our review. The articles, sourced from five databases, were peer-reviewed and published from January 2000 to April 2023. We found that complexity is considered an influencing factor in procurement, but usability is not addressed broadly during procurement, although complexity is perceived as affecting usability. There is a lack of appropriate methods for evaluating usability while procuring complex IT systems. This gap stems from varying priorities in usability evaluation. Procurement prioritizes lightweight assessment and comparability, whereas evaluation of complex systems requires a comprehensive methodology and a broad perspective. Based on our findings, we propose that complexity needs to be evaluated within the context of usability during the procurement of complex IT systems by considering the three aspects of complexity: task, system, and context.
Keywords
usability, procurement, complexity, IT system, complex system
Introduction
The acquisition of large information systems for use in varying contexts, such as healthcare and education, poses a continuing challenge. Common difficulties include proper definitions of requirements, technical implementation, and integration of systems into the organization. Such problems become more complex as the number of interested stakeholders and the functionality required of the system increase. In the case of public and government systems, the outcomes of such projects and procurement often generate broader interest and discussion. Acquisition of large systems poses a continuing challenge in varying contexts such as healthcare and education. Common difficulties include proper definitions of requirements, technical implementation, and integration of systems. Such problems amplify as the number of interested stakeholders and the functionality required of the system increase. Due to these factors, there appears to be a disconnection between the process to procure systems and ensuring those systems will be usable for the end user. In order to address this disconnect, processes and activities need to be modelled and developed. To do so, focus in research and practice is required. Procurement, especially public procurement, is an important process needing improvement.
Procurement is the process through which organizations buy goods and services. Procurement can also be thought of as an instrument for achieving organizational change because large IT systems affect multiple organizational processes and work practices (Po-An Hsieh & Wang, 2007). The need for a new IT system often arises from the desire to enable new and more efficient ways of working. For the new IT system to generate financial benefits or save resources, it must function properly and fit its intended purpose. The end users must be able to complete their tasks with it. Hertzum (2021) reflects on previous studies with results that show organizational change is sociotechnical in nature. Implementing new ways of working requires more than technology; it entails adoption at both management and user levels. If users do not adopt a new system, both that system and its deployment in the organization will fail, leading to a situation in which the planned benefits will not be realized.
Although the procurement process can be considered complete once the new system is set up, the whole process will amount to nothing if users are unable to do their work with the system. This perspective on procurement highlights the importance of considering usability, user experience, and accessibility throughout the process.
Potential service providers submit their proposals to the procuring organization during the tendering process. The tendering process, by design, weighs the advantages and disadvantages of different options offered by competing vendors; comparison is inherent. For functional requirements, this process may be straightforward. Does a system fulfill the required function or not? For quality requirements, such as usability, comparability becomes more abstract. Requiring the “system [to] be usable” is difficult to express in numerical terms, whereas a rudimentary requirement such as “the user must be able to fill form x in timespan y” can be roughly evaluated in terms of whether it is completed or unfinished (Carey, 1991).
Despite the apparent importance of acquiring useful and usable software, public-sector challenges with usability in large organizations with complex IT systems seem to generate constant discussion and even headlines in the media. This can be partly traced back to managing usability requirements in procurement (Lehtonen et al., 2010). As such, these problems are not limited to usability requirements but seem to apply to requirement specifications in general, which pose the top major challenge in procurement, as pointed out by procurement specialists in Norway (Moe & Päivärinta, 2011). Existing literature regarding usability in procurement appears limited, but the importance of non-functional quality requirements, including usability, has been highlighted (for example, Carey, 1991; Lauesen, 1998).
Background on Procurement
Different regulations apply to the procurement process, depending on whether the procuring organization belongs to the public or the private sector (Strand et al., 2011). While regulations vary among countries, some basic principles can be defined. The procurement phases can be generalized as follows: Identify a need and define the requirements; Perform a market analysis; Submit a call for tenders; Evaluate and select vendors; Negotiate the terms, and; Finalize the purchase (Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on Public Procurement and Repealing Directive 2004/18/EC, 2014; Novack & Simco, 1991). Specific regulations apply to public organizations to ensure a transparent and unbiased process, for example, in the European Union (EU) (Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on Public Procurement and Repealing Directive 2004/18/EC, 2014) and the USA (Federal Acquisition Regulation [FAR], 2019). The EU legislation also defines different types of procurement procedures, providing a good example of a typical procurement project (Figure 1). A call for tenders can include both functional and non-functional requirements.
Figure 1. Open tendering procedure (adapted from Moe et al., 2017).
Some studies regarding usability in procurement have been published. In one of the earlier case studies, Artman (2002) found that the procurer and the vendor were unable to discuss usability in general because they had completely different understandings of the topic and its related goals. This difference in interpretation was partly attributed to the varying positions and areas of expertise of the people involved in the procurement process. A decade later, Lehtonen et al. (2010) reported the poor state of usability requirements in procurement; out of more than 40 analyzed cases, usability requirements were found in six categories, but not a single call-for-tender included true usability requirements. The researchers noted a few cases in which usability requirements were identified but placed under the category of “the system must be usable.”
While the findings by Artman (2002) and Lehtonen et al. (2010) did not provide a positive outlook regarding the state of usability in procurement, Riihiaho et al. (2015) reported successfully incorporating usability testing in the public tendering process. By defining the minimum level of usability of candidate systems and then testing it, they were able to make informed and justifiable decisions regarding the selection of a complex IT system (Riihiaho et al., 2015). Moe and Päivärinta (2011) described the findings on organizational change in IT procurement in their panel discussions. For instance, in the public sector, the “change of work processes and benefits realization” was regarded as one of the top procurement challenges (Moe & Päivärinta, 2011, p. 411).
Complexity of IT Systems
Originally used in ecology, the term complexity has later been applied to numerous fields (Turner & Baker, 2019). According to Po-An Hsieh and Wang (2007), complex IT systems are large organizational systems that streamline business processes across various functional departments. This definition brings into focus the importance of considering usability during procurement. Based on their definition, organizational change and the ways of working are interlinked with IT systems; if a complex system is defined as “integrating and streamlining business processes,” acquiring new systems must affect such processes (Po-An Hsieh & Wang, 2007, p. 216).
In the usability context, complexity can be examined through the lens of either the system or the work being done. Zou et al. (2018) investigated how complexity manifested in product–service systems. They outlined four aspects: multiplicity, diversity, interdependence, and variability. All these factors describe the states of units, subsystems, and actors related to the product–service systems and thus do not fall under the categories of either work or system but provide a framework for both.
Mirel (2004) presents the idea of complex problem solving and how this type of work fundamentally differs from linear work in which tasks are easy to define and results are easy to evaluate: Complex problem solving involves large amounts of data, uncertainty, and high-order analyses that require interpretation and sensemaking of relationships in the data. This idea has been further explored by Redish (2007), who claims that the nature of complex problem-solving evidently leads to the fact that the usability of such work cannot be evaluated using traditional methods, which often rely on linear and well-defined tasks. Referring to both Mirel (2004) and Redish (2007), Albers (2011) argues that, in the usability context, examining how complexity manifests in practice is more beneficial than its exact definition. Still (2011) proposes that an efficient way to acknowledge complexity is by examining the usability context.
These concepts by Mirel (2004) and Redish (2007) can also be found in industrial fields where the processes do not only involve singular IT systems but a wide array of other instruments. This kind of system-wide usability has been studied in control-room environments of maritime operations (Pan et al., 2015) and nuclear power plants (Savioja & Norros, 2013). In both instances, the authors acknowledge that to evaluate the usability of the whole system, it is not enough to evaluate only singular parts.
It seems that complexity has not gained a lot of discussion in the usability field. While it is impossible to determine the exact reason for this, the wide array of definitions and the abstract nature of complexity might play a role. To study and discuss the relation between complexity and usability, it is useful to examine the associated characteristics. Three common aspects relevant to the often-used definitions of usability can be identified from these ideas of complexity in relation to IT systems. These aspects are task, system, and context (Hertzum, 2021; International Organization for Standardization, 2018; Nielsen, 1994). Task refers to the complexity of users’ work. System pertains to the complexity of the technical IT system and how that complexity affects its use. Context considers aspects in user’s surroundings that are not directly connected to the system or tasks.
Aim of the Article
In this paper, we explore how usability has been and can be considered in the procurement of complex IT systems. We focus on the procurement process and the integration of usability aspects into it, such as end-user task definitions, usability requirement elicitation, and usability evaluation. According to the studies conducted by Mirel (2004) and Redish (2007), who address usability and complex problem solving, to successfully evaluate the usability of complex IT systems, the methodology must be reconsidered. Therefore, a complex IT system’s usability cannot be evaluated by purely examining simple linear tasks and use cases. In this article, we approach our research question from two perspectives: 1) a case study of a public organization’s experiences with integrating usability into the procurement of complex IT systems and 2) a scoping review of how complexity has been acknowledged in the existing academic literature regarding usability in procurement. Our aim is to increase the understanding of complexity and how usability can be considered in practice when procuring complex IT systems.
Methods
This research consists of two parts: a case study followed by a scoping review. Although the order of the research we conducted is uncommon, our initial target was to briefly review the background literature and then study how these issues had been addressed in practice. However, after the initial background search was done for the case study, we realized that we needed to conduct a broader scoping review on usability and complexity in procurement to ensure we hadn’t missed anything.
Case Study
We conducted an empirical study to approach the research question from a practical viewpoint. Our aim was to research IT professionals’ experiences and views about complexity and usability in IT procurement.
Participants and Study Procedure
We selected a Finnish university’s internal IT unit for the case study. Their staff are experienced in complex IT system procurement. We used semi-structured interviews, which allowed exploration and improvisation in the context of a multifaceted topic in which the experiences and knowledge of the interviewees might differ (Runeson & Höst, 2009). We recruited the participants based on the following criteria: 1) experience with procurement, 2) worked on an IT system in some capacity at the organization, and 3) able to describe how the procurement processes within the organization had been developed over time based on each participant’s own experience. We decided not to restrict the criteria to a specific role, such as architect, procurer, or developer, because we knew in advance that finding suitable participants would be difficult due to the limited total number of possible participants at the organization. We wanted to interview people with diverse backgrounds regarding the same topic, predicting that professionals would have varying conceptions of abstract themes such as complexity.
The semi-structured interview questions revolved around three themes: complexity, usability, and procurement. The interview guide was used to steer the conversation by first asking background questions regarding the participant’s position at the company, then asking questions on the themes, including leading questions and more detailed questions (Table 1). We conducted a pilot interview before the actual interviews to ensure sufficient clarity and structure of the questions.
Table 1. Interview Themes and Main Questions
| Topic | Questions |
|---|---|
| Complexity | In your own words, what is the IT environment like at this organization? |
| What do you think: Does the complexity of the organization’s IT systems affect their usability? | |
| Usability | How would you describe the usability of the organization’s IT systems in general at the moment? |
| What do you think are the biggest challenges regarding usability in your current work? | |
| Procurement | In the procurement projects in which you have participated, how has usability been taken into account during tendering? |
| Do you think that the definition of usability requirements for tendering purposes has led to a better outcome? |
One of the authors (TS) conducted the interviews in August 2022. The research data consisted of interviews with seven IT professionals working at the organization. All participants had over a decade of experience in varying roles at the organization. Their roles were related to the overall IT architecture, management of IT functions at the organization, or development and maintenance of a complex IT system. Five of the seven interviewees had previous experience with procurement within the organization, and the other two had worked directly on implementing and developing the procured major IT systems. One researcher conducted all interviews remotely via Microsoft™ Teams™ while video recorded and transcribing them for the data analysis.
Data Analysis
The data analysis followed the principles and seven steps of the thematic analysis method (Braun & Clarke, 2006). The analysis can be described as a mix of inductive and deductive approaches (Skjott Linneberg & Korsgaard, 2019) due to its distinct areas of focus. Initially, we categorized 413 codes into 18 themes, which included some overlapping codes. Next, we reviewed and named the themes by utilizing affinity diagramming (Beyer & Holtzblatt, 1997; Hartson et al., 2001). The final iteration of the analysis, in which we merged the overlapping codes and similar themes, included six themes in total: 1) complexity, 2) system-wide usability, 3) users’ input, 4) organizational processes, 5) usability in procurement, and 6) system usability.
Scoping Review
We conducted a scoping review after the case study to better understand whether complexity has been discussed in existing research that addresses usability in procurement. The review process mainly followed the PRISMA guidelines (Tricco et al., 2018). To supplement our main research question (how can usability be considered in the procurement of complex IT systems), we focused on the following sub-questions:
- Has complexity (or have some aspects of it) been mentioned in the article?
- Has complexity (or have some aspects of it) been recognized as a factor influencing procurement?
- Has complexity (or have some aspects of it) been recognized as a factor influencing usability or UX?
- How has complexity been considered in procurement? Are there recommendations
Search and Selection of Articles
We searched for peer-reviewed academic articles (published from January 2000 to April 2023) that discussed the role of usability in procurement. We chose this time range because IT environments, IT systems, and their procurement have changed over the past two decades. We decided to include “UX” as a search term in addition to “usability” because the terms are closely related, and in some articles, the terms are used in parallel. Furthermore, we wanted to ensure that we would find all relevant articles for our review despite the terminology used.
Table 2. Inclusion and Exclusion Criteria for the Scoping Review
| Inclusion criteria | Exclusion criteria |
|---|---|
| Article written in English | Does not discuss procurement |
| Published from January 2000 to April 2023 | Does not discuss usability or UX |
| Peer-reviewed, sufficiently long (more than two pages) academic article (published in a journal or in conference proceedings) that includes academic references | Does not discuss procurement of IT systems or software |
| Includes the term(s) procurement, selection, and/or acquisition | |
| Includes the term(s) usability and/or UX |
Three researchers (MT, KS, and JV) searched four databases (ACM Digital Library, IEEE Explore, Scopus, and ScienceDirect) from February to April 2023. We also searched Google Scholar™ to see if we could find additional articles.
In the identification phase, our search targeted titles, abstracts, and keywords. The main search terms were “(procurement OR acquisition OR selection) AND (usability OR UX)”. However, for cases in which the first search produced over 400 results, we narrowed the search either by focusing on the title or adding the search string “system OR service OR IT”. There were variations due to some differences in the search options in the databases, such as when the search could only be conducted in full text or by title. Table 3 lists the exact search phrases per database and the resulting number of articles during the identification phase.
Table 3. Search Strings and Results in Each Database
| Source | Search string 1 | Results | Search string 2 | Results |
| ACM Digital Library | Title, abstract, keywords, 2000–2023(02): (procurement OR acquisition OR selection) AND (usability OR UX) | 46 | ||
| IEEE Explore | Title, abstract, 2000–2023: procurement AND (usability OR UX) selection AND (usability OR UX) acquisition AND (usability OR UX) | 745 | Title, abstract, 2000–2023: procurement AND (usability OR UX) selection AND (usability OR UX) acquisition AND (usability OR UX) AND the title includes ‘system’ OR ‘IT’ OR ‘software’ | 175 |
| Scopus | (TITLE-ABS-KEY ( (procurement OR acquisition) AND (usability OR UX) ) AND PUBYEAR > 1999) AND TITLE ( (system OR software OR IT) ) AND (LIMIT-TO (PUBSTAGE, “final”) ) AND (LIMIT-TO (DOCTYPE, “cp”) OR LIMIT-TO (DOCTYPE, “ar”) OR LIMIT-TO (DOCTYPE, “ch”) ) AND (LIMIT-TO (LANGUAGE, “English”) ) | 426 | (TITLE-ABS-KEY (usability OR UX) AND TITLE (selection) AND PUBYEAR > 1999) AND TITLE ( ( system OR software OR IT) ) AND (LIMIT-TO (PUBSTAGE, “final”) ) AND (LIMIT-TO (DOCTYPE, “cp”) OR LIMIT-TO (DOCTYPE, “ar”) OR LIMIT-TO (DOCTYPE, “ch”) ) AND (LIMIT-TO (LANGUAGE, “English”) ) | 66 |
| Science Direct | (procurement OR acquisition OR selection) AND (usability OR UX) | 1 | ||
| Google Scholar | All in title: (procurement OR acquisition OR selection) AND (usability OR UX) AND (system OR software OR IT) | 64 |
In Scopus, we conducted two separate searches, but in IEEE Explore, we conducted the second search based on the results of the first search.
After obtaining a list of articles from each source, we listed the articles in a spreadsheet. Then, during the screening phase, we reviewed the articles by their titles and placed each under one of the three categories: yes, no, or maybe. The three researchers then discussed the articles under the maybe category and either included them in the following stage or excluded them based on their titles. Next, removing the duplicates resulted in 106 articles that were subsequently reviewed, based on their abstracts, in the eligibility phase. The screening stages are presented in Figure 2.
Figure 2. The different stages of the scoping review’s selection process based on each database.
In the eligibility phase, the 106 articles were divided among the three researchers for the abstract review. During this phase, we selected the articles that discussed usability or UX in software or IT system procurement. In unclear cases, we again discussed the abstracts and decided whether they should be included in or excluded from the next stage (selection for full-article analysis), which yielded 25 articles. During this eligibility phase, we also discovered three additional relevant articles, resulting in a total of 28 articles.
The three researchers participated in reviewing the full articles. Two of the three researchers read each of the 28 articles left after the screening process. Both researchers answered the research sub-questions independently and then discussed and combined their results. The final list of the 28 reviewed articles is in Appendix A.
Results
Case Study
We present the case study results under six themes: complexity, system-wide usability, users’ input, organizational processes, usability in procurement, and system usability.
Complexity
The participants had differing views on the nature of complexity in the organization. All participants agreed that the IT environment at the organization could be considered complex, but they had varying reasons for this. The more technically oriented participants described factors such as the number of systems in use and the mixing of legacy systems with new systems. In contrast, the participants in managerial roles described complexity as resulting from the size of the organization, the diverse needs of the stakeholders, and the varying processes leading to the procurement of IT systems.
When considering the use of the terms complex versus complexity in the interviews, we first need to highlight that the interviews were held in Finnish, so there might be some misalignment with the English terminology. There was no general definition of complexity that would have emerged from the interviews or a single item that was highlighted when describing the complexity of the environment. The interviewees sometimes used complexity as a synonym for complicated, but in general, this was not the case. Rather, they described similar aspects as the literature: number of systems or users, organization, and time. Notably, complexity was very rarely mentioned in the context of users’ tasks.
System-Wide Usability
When discussing usability from the perspective of all systems required to complete work tasks, the participants uniformly described the overall experience at the organizational level as lacking in quality. They identified specific problems, such as the difficulty in choosing the correct tool for a task and the limitations imposed on usability by legislation regarding security. A notable limitation of system-wide usability was the interaction of ready-made systems. More precisely, a participant identified the problem as the varying logics of the user interfaces in different systems. Some interviewees wondered whether other existing systems should be considered when procuring a new IT system. One of the technically inclined participants discussed how such cross-functionality and integration were considered quite extensively, in a purely technical sense, such as in the case of data flow. The participants mentioned that making similar considerations for usability would be quite difficult. However, they could not elaborate as to why because this had not been done previously at the organization.
Users’ Input
The discussion regarding new systems and ensuring usability indicated a lack of user involvement during procurement. Some interviewees perceived this as a fault on the organization’s part, meaning that users’ input should be considered more strongly during procurement. In contrast, an interviewee expressed the opinion that the organization should not directly involve singular users in such processes because the interviewee believed that the users might have opinions that did not reflect the real needs of the wider user base. Most participants thought defining the users’ tasks and work processes was straightforward, but mapping and documenting them was not done systematically at the organization. Regarding users’ input, an interviewee who had participated in implementing a procured complex IT system mentioned that the specifications at the procurement phase consisted of a minimum viable product, and it was known from the start that most of the user paths and features would be implemented after procurement. In this context, the participant explained that they believed that users’ needs would be at least partly carried over because implementing the new system mainly focused on replicating the features of the old one being replaced. The participant elaborated that the previous system had been iterated and improved for a very long time according to the users’ wishes and that even though the system was new, the use cases were not.
Organizational Processes
The discussions directly related to this theme brought up points regarding organizational change and how acquiring a new IT system could be an opportunity to re-evaluate and change an organizational process. However, the participants also mentioned that IT systems could impose limitations on how certain processes should be done. While this theme came up least among the six, it was implied in the context of other themes. For example, multiple interviewees perceived the number of differing processes for similar tasks within the organization to be a source of complexity and a limitation to system-wide usability.
Usability in Procurement
The participants did not regard usability in the procurement process as a priority. According to them, including usability requirements had the main advantages of mandating the vendor to provide a demonstration and allowing the procuring organization to disregard vendors that failed to demonstrate even the “base level of usability” of the solution. Additionally, the participants often mentioned that the solutions being purchased were often customizable, meaning that it would be easier to buy a platform that could cover multiple needs and, after the strenuous procurement process, hone the exact features and use cases. With both aspects, the participants stated that the procurement process consumed a lot of resources, as mentioned by one who had played a key role in multiple procurement projects: “I don’t think we have the resources to define or analyze usability. We need external evaluators. Most of us are not usability experts.” For the organization, it was more beneficial to complete the process as quickly as possible and iron out the details later.
System Usability
This theme mainly consisted of findings on how usability evaluation was conducted during procurement at the organization. According to the participants, in most cases, the process included heuristic evaluations of approximately three main use cases described in the request for tenders. External evaluators conducted these. The procurement team defined the use cases in tandem with the organizational department for which the upcoming IT systems were being acquired. An interviewee with extensive experience in procurement projects said that, in most cases, end users were not consulted when the use cases were being defined. When discussing the roles of usability and complexity in procurement, one participant claimed, “…the most important thing is the minimum-level [usability] so that we don’t need to pick the cheapest option. Especially if [the candidate system] is unusable. And so, in the overall evaluation, if it has better usability, it can be more expensive.” By this, the interviewee pointed out, if there are no quality criteria for usability, the price might be the only deciding factor.
Scoping Review
We present the results of our scoping review under four themes: complexity or its aspects, complexity as a factor influencing procurement or usability, considerations of complexity in previous studies, and recommendations for considering complexity in procurement. A synthesis of the review follows.
Complexity or Its Aspects
Of the 28 reviewed articles, 12 mentioned the term “complexity.” In 13 articles, some aspects of complexity (task, system, and context) were mentioned, although complexity as a term might not have been discussed by all. Eight articles (Artman & Zällh, 2005; Borycki & Kushniruk, 2023; Dawood et al., 2023; Grier, 2013; Krause & De Lusignan, 2010; Kushniruk et al., 2010; Lee et al., 2016; Tyllinen et al., 2016) discussed complexity, procurement, and usability. Four articles (Cajander et al., 2007; Carvallo et al., 2003; Fumagalli et al., 2019; Hardless & Jaffar, 2011) in some way addressed all of these but did not discuss their relations. One article (Markensten & Artman, 2004) discussed some aspects of complexity without specifically mentioning complexity. Table 4 summarizes which articles discussed complexity or one of its aspects.
Table 4. Articles Discussing Complexity
| Topic | Articles |
|---|---|
| Complexity | Artman and Zällh (2005); Borycki and Kushniruk (2023); Cajander et al. (2007); Carvallo et al. (2003); Dawood et al. (2023); Fumagalli et al. (2019); Grier (2013); Hardless and Jaffar (2011); Krause and De Lusignan (2010); Kushniruk et al. (2010); Lee et al. (2016); Tyllinen et al. (2016) |
| Aspect: Task | Artman and Zällh (2005); Cajander et al. (2007); Kushniruk et al. (2010); Tyllinen et al. (2016) |
| Aspect: System | Artman and Zällh (2005); Cajander et al. (2007); Fumagalli et al. (2019); Grier (2013); Hardless and Jaffar (2011); Krause and De Lusignan (2010); Kushniruk et al. (2010); Lee et al. (2016); Markensten and Artman (2004); Mehdiyev (2022); Tyllinen et al. (2016) |
| Aspect: Context | Artman and Zällh (2005); Cajander et al. (2007); Dawood et al. (2023); Hardless and Jaffar (2011); Krause and De Lusignan (2010); Kushniruk et al. (2010); Lee et al. (2016); Markensten and Artman (2004); Tyllinen et al. (2016) |
Grier (2013) noted that “complex systems” might be connected to others offered by different providers. Fumagelli et al. (2019) concluded that in complex systems, it would be more important to find one that fit the organization’s goals rather than the system with the best technical performance.
Grier (2013) discussed complex tasks, for example, stating that to perform one task, a user might need several systems developed by different providers. Tyllinen et al. (2016) noted that complex scenarios (tasks) would require experiences with the candidate systems that could be reported.
In the articles, discussions about complexity referred to the complexity of the systems and the complexity of usage environments. The articles represented a broad range of disciplines and sectors. The most often discussed sector was healthcare (Borycki & Kushniruk, 2023; Krause & De Lusignan, 2010; Kushniruk et al., 2010; Lee et al., 2016; Tyllinen et al., 2016), alongside examples of military systems (Grier, 2013), a taxi dispatch system (Artman & Zällh, 2005), and a learning management system (LMS) (Dawood et al., 2023).
When considering the healthcare sector, complexity was discussed in several contexts. Kushniruk et al. (2010) discussed the complex hospital environments and their complex workflows, as well as the organizations’ complexity. They also acknowledged the complexity of usage scenarios. Krause and De Lusignan (2010) noted the complexity of the clinical environment, as well as the IT system’s complexity and how it affected the decision-making process in procurement, making it difficult to involve the user community. Tyllinen et al. (2016) recognized how usability evaluation of complex IT systems affected the procurement of these systems in healthcare. They also mentioned the need for more cost-efficient approaches that would balance the variety of usage scenarios, user roles, and workflows that need to be evaluated with the limited resources in procurement. Furthermore, Lee et al. (2016) explained the complexity of joint procurements of several organizations (in the context of hospital systems, for example, electronic health record [EHR] systems) and the complexity of the IT systems.
Regarding other contexts, Grier (2013) described the procurement process of military systems and the systems themselves as complex. They stated that the operational test-and-evaluation phase of the procurement process was less controlled and more complex than a standard usability evaluation (for example, when utilizing standardized usability questionnaires). Dawood et al. (2023) identified that often complex systems are used by many different user groups whose perspectives on the system differ and need to be considered in the selection. They argued that the selection process of such systems, in their case an LMS, is a multicriteria decision making problem, and proposed a usability evaluation framework that takes these different user perspectives into account when weighing and ranking the alternatives.
Complexity as a Factor Influencing Procurement or Usability
In general, only a few articles explicitly discussed complexity as a factor influencing procurement or usability. Artman and Zällh (2005) stated that complexity affected usability. Lee et al. (2016) noted that the complexity of the procurement process and the contract limitations were perceived as influencing usability.
Considerations of Complexity in Previous Studies
At a more general level, we examined how complexity was considered in the articles. We found that in studying the evaluation of industrial simulation software, Fumagalli et al. (2019) noted that the tool’s level of complexity was perceived as representing the complexity of the production system. In the context of military environments, Grier (2013) noted that military systems comprised complex systems connected with other complex systems.
Regarding how usability had been addressed in the context of complex systems, complexity was viewed as a factor that posed challenges by affecting the system’s usability, making usability work more difficult, or disregarding users when procuring or developing complex systems. Lee et al. (2016) identified complexity as a factor that made it more difficult to produce a usable system. Kushniruk et al. (2010) recognized EHR systems, work tasks, and organizational needs as complex, which affected the system’s usability and the procurement process. Borycki and Kushniruk (2023) observed that a centralized certification of a vendor product did not necessarily indicate a system as learnable, easy to use, efficient, and safe when deployed in a healthcare setting, and Krause and De Lusignan (2010) acknowledged that decisions on procuring a complex system were made without contact with the users and that the requirements could not be changed. Tyllinen et al. (2016) pointed out the lack of methods for evaluating the usability of complex systems, which affected the procurement process. Dawood et al. (2021) noted that complex systems had many types of users and addressing one’s perspective inevitably challenged those of other user types, leading to an ill-advised decision in the selection process.
On the positive side, Cajander et al. (2007) reported that the analysis techniques intended for commercial off-the-shelf systems were also suitable for complex systems when the context of use was acknowledged. Tyllinen et al. (2016) brought up the complexity of both the environment and tasks as elements of the usability evaluation method that they had developed.
Recommendations for Considering Complexity in Procurement
Our last research questions for the scoping review focused on the recommendations related to complexity, procurement, and usability. Previous research pointed out usability as important for the smooth operations of a complex system (Artman & Zällh, 2005). While technical certification does not assure usability, human factors should be considered during procurement (Borycki & Kushniruk, 2023). Kushniruk et al., 2010 also recommended an evidence-based inspection system instead of a demonstration in front of a panel, which is a common practice, at least in healthcare systems’ procurements. Lastly, Dawood et al. (2021) proposed a usability evaluation framework for scoring several criteria in software selection.
When developing complex IT systems, several evaluations are needed to create a usable system, and usability must be considered at an early stage (Artman & Zällh, 2005). Additionally, the focus should be on the use cases of the system, instead of its functions (Krause & De Lusignan, 2010) and on written specifications (Cajander et al., 2007).
Future research should pay attention to intra-organization-related aspects (Hardless & Jaffar, 2011) and new methods for evaluating usability in procurement (Tyllinen et al., 2016).
Synthesis
To summarize our scoping review, we observed that usability and complexity had seldomly been addressed in the context of procuring IT systems, and even then, not in a thorough way. Some articles (Artman & Zällh, 2005; Borycki & Kushniruk, 2023; Dawood et al., 2023; Grier, 2013; Krause & De Lusignan, 2010; Kushniruk et al., 2010; Lee et al., 2016; Tyllinen et al., 2016) addressed this combination and noted that usability should be considered when procuring complex IT systems (Artman & Zällh, 2005; Borycki & Kushniruk, 2023). Some researchers also pointed out that it would require special attention in the evaluation process ( Artman & Zällh, 2005; Kushniruk et al., 2010, Dawood et al., 2023) as well as modified usability evaluation methods (Tyllinen et al., 2016).
Discussion and Conclusions
The general consensus is that, when discussing complexity in relation to usability, complexity often indicates increased difficulty due to the large number of related factors and aspects requiring consideration. Earlier research suggested that in some cases, the system, the tasks, or the environment had been noted as complex (Artman & Zällh, 2005; Grier, 2013; Kushniruk et al., 2010); however, the relationship among these aspects had not been broadly considered. Given the increasing complexity of IT systems and challenges in the successful procurement and implementation of such systems in organizations, our aim was to research how usability had been and could be considered in the procurement of complex IT systems.
Main Findings
Based on the findings of our study, usability has not been widely considered in the context of procuring complex IT systems. Based on our scoping review, the complexity of both the system and the context have been identified as relevant factors when procuring an IT system. During procurement, complexity has been taken into account in a limited way in relation to usability and only in contexts involving regulation and clearly defined work processes. Only one article (Artman & Zällh, 2005) directly considered complexity as a factor affecting usability. In other articles that took complexity into account, this relationship was not explicitly stated but implied by focusing on specific aspects, such as the context of the system. Interestingly, all the procurement experts in our case study’s organization were ready to claim that complexity affected usability.
Another important finding was the lack of approaches and appropriate methods for evaluating the usability of complex IT systems during procurement. Our case study highlighted the role of usability in procurement from a limited perspective. This question arises: Will usability evaluation during procurement provide valid results in relation to the outcome of the procurement if complexity has not been considered? Furthermore, how can complexity be truly considered if using an elaborate and extensive usability evaluation methodology (as complex systems require) is not necessarily compatible with the procurement process?
Although the case study and some of the articles in the scoping review acknowledged the connection between complexity and usability, our case study suggests that the role of usability requirements in tendering is not intended to ensure that the quality of the acquired product is as good as it can be. Instead, usability requirements and evaluations are applied to establish a “base level” of usability, which can then be used to disregard vendors who are unable to produce a demonstration. The role of usability is diminished at this stage and postponed until implementation. However, when acquiring ready-made IT systems, the system development phase is completed before procurement. This brings up the question of whether this strategy is viable.
To accurately evaluate the usability of a complex system in procurement, including the context and the tasks related to it, the approach used in the evaluation should more thoroughly consider all three aspects of complexity –task, system, and context. Based on our findings, such a comprehensive and reliable usability evaluation approach to complex IT systems seems counterintuitive to other procurement goals.Existing literature confirms this contradiction; traditional usability evaluation methods do not seem to support evaluating complex IT systems (Redish, 2007). Usability evaluation in a complex context, such as that of industrial control rooms, requires the development of an extensive and elaborate methodology (Savioja, 2014). These heavy and time-consuming methods are unsuitable for procurements, where cost-effective methods are needed. The traditional usability evaluation methodology and the process for including these evaluations in procurement have been further developed for the context of health and social welfare IT systems (Tyllinen, 2023). To evaluate complex IT systems in a manner that takes into account the actual use environment and users’ real-life work, a more extensive methodology should be considered and developed (Pan et al., 2015; Savioja & Norros, 2013; Still, 2011) also for procurement (Tyllinen, 2023). In this study, we also pointed out that during procurement, when usability evaluation is used to compare systems (Dawood et al., 2023; Tyllinen et al., 2016), the methodology should be kept as light and efficient as possible. What the exact methods would be that suit this context (procurement of complex IT systems) remains an area for future research.
Reflection: Aspects of Complexity
Based on the background literature, we identified three aspects of complexity relating to usability: task, system, and context (Hertzum, 2021; International Organizatizon for Standardization, 2018; Nielsen, 1994). The complexity of the users’ work is captured by the aspect of “task,” which entails complex problem-solving, lack of structure, and complexity of information. “System” describes the technical complexity of the IT system environment affecting its use. “Context” involves factors outside the other two, such as social relationships and the physical environment. Based on the literature, the domains that consider the complexity of tasks are highly regulated, namely the military and healthcare, where work processes are well defined by the necessity of adhering to regulations. Based on existing literature, our categorizations of complexity and usability suggest that the concepts have some common characteristics. Figure 3 presents the three aspects of complexity and examples of what kinds of issues increase complexity. It also includes issues of how they have been perceived in procurement based on our scoping review. In Figure 3, we also mapped three examples of domains: healthcare, the military, and the case study (representing an educational environment), which are complex for different reasons. In healthcare, tasks and the context are complex, whereas in the case study of an educational environment, complexity is caused by the system and the context. In the military environment all, the task, system and context can be complex.
The usability of complex systems can be viewed through these three aspects. In practice, understanding these aspects can help focus usability evaluations during procurement and define user requirements for the relevant aspect of complexity. This helps recognize the most significant use cases and brings them into the evaluations during procurements. Thus, users’ broader goals can be considered instead of focusing on distinct user tasks and activities. Furthermore, the requirements caused by the context can be better identified as related to complexity. To conclude, utilizing these three aspects can help recognize the meaningful larger entities that can be used to focus human-centered design activities. The concrete activities and methods need to be concentrated on in future research.
Figure 3. Aspects of complexity mapped to the examples of domains.
Limitations
Our study’s limitations are mostly related to its scope. Contrary to the usual sequence, we conducted the case study before the scoping review. This approach led to minor terminology inconsistencies, such as the phrase “system-wide usability.” Additionally, the aspects of complexity, which we employed in our analysis of the scoping review, originated partly during the case study. For this reason, our thematic discussions with the IT professionals did not touch on the aspects of complexity. After our scoping review, it also became clear that this research would have benefited from a larger sample size of organizations.
Conclusion
We conducted a case study and a scoping review to understand usability during the procurement of complex IT systems. We discovered that complexity has been considered a factor influencing procurement, but usability has not been broadly addressed, although complexity is perceived as affecting usability. There is a lack of appropriate methods for evaluating usability when procuring complex IT systems. The reason might arise from differing priorities between usability evaluation in procurement and usability evaluation of complex IT systems. The former prioritizes lightweight evaluation and comparability, whereas the latter often requires extensive methodology and a broad scope. Based on our findings, we propose that complexity, in the context of usability, should be evaluated by considering three aspects of complexity: task, system, and context. We have mapped the issues affecting usability and procurement regarding these three aspects.
Tips for Usability Practitioners
- Technical compatibility with the existing IT architecture does not guarantee that the new system will have high usability or provide a pleasant experience for the end users.
- Including well-defined usability requirements in tendering can provide a tangible basis for comparative evaluation, especially in the context of public procurement processes.
- Acknowledging only linear and easily definable use cases might lead to a lack of understanding about the usability of a complex IT system.
- The relevant aspects of complexity should be recognized for each case to help identify and focus on the most significant use cases when determining the requirements and constructing usability evaluations for the procurement process.
- Usability practitioners should consider the existing IT architecture and appropriate methods for using evaluation tools when defining usability evaluation criteria for the procurement of complex IT systems.
References
Albers, M. J. (2011). Usability of complex information systems. In M. J. Albers & B. Still (Eds.), Usability of complex information systems: Evaluation of user interaction (pp. 3–16). CRC Press.
Artman, H. (2002). Procurer usability requirements: Negotiations in contract development. Proceedings of the Second Nordic Conference on Human-Computer Interaction, 61–70. https://doi.org/10.1145/572020.572029
Artman, H., & Zällh, S. (2005). Finding a way to usability: Procurement of a taxi dispatch system. Cognition, Technology and Work, 7(3), 141–155. Scopus. https://doi.org/10.1007/s10111-005-0182-6
Beyer, H., & Holtzblatt, K. (1997). Contextual design: Defining customer-centered systems. Morgan Kaufmann Publishers Inc.
Borycki, E. M., & Kushniruk, A. W. (2023). Human factors in healthcare IT: Management considerations and trends. Healthcare Management Forum, 36(2), 72–78. Scopus. https://doi.org/10.1177/08404704221139219
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Cajander, Å., Eriksson, E., & Gulliksen, J. (2007). Evaluating procurement, usability and off-the-shelf office software. In E. L.-C. Law, M. K. Lárusdóttir, & M. Nørgaard (Eds.), Downstream utility: The good, the bad, and the utterly useless usability evaluation feedback (pp. 25–29). Institute of Research in Informatics of Toulouse (IRIT).
Carey, T. T. (1991). A usability requirements model for procurement life cycles. In J. M. Carey (Ed.), Human factors in information systems: An organizational perspective (pp. 89–104). Ablex.
Carvallo, J. P., Franch, X., & Quer, C. (2003). Defining a quality model for mail servers. In H. Erdogmus, & T. Weng (Eds.), Lecture notes in computer science: Vol. 2580. COTS-based software systems (pp. 51–61). Springer. https://doi.org/10.1007/3-540-36465-X_5
Dawood, K. A., Zaidan, A. A., Sharif, K. Y., Ghani, A. A., Zulzalil, H., & Zaidan, B. B. (2023). Novel multi-perspective usability evaluation framework for selection of open source software based on BWM and group VIKOR techniques. International Journal of Information Technology & Decision Making, 22(01), 187–277. https://doi.org/10.1142/S0219622021500139
European Parliament and the Council. (2014). Directive 2014/24/EU on public procurement and repealing Directive 2004/18/EC (Directive 2014/24/EU [February 26, 2014]). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A02014L0024-20220101
Federal Acquisition Regulation (FAR). (2019). Title 48—Federal acquisition regulations system: Chapter 1 [CFR]. https://www.acquisition.gov/sites/default/files/current/far/pdf/FAR.pdf
Fumagalli, L., Polenghi, A., Negri, E., & Roda, I. (2019). Framework for simulation software selection. Journal of Simulation, 13(4), 286–303. https://doi.org/10.1080/17477778.2019.1598782
Grier, R. A. (2013). The potential utility of the system usability scale in U.S. Military acquisition. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57(1), 206–209. https://doi.org/10.1177/1541931213571046
Hardless, C., & Jaffar, A. (2011). Heterogeneous inter-organisational IT innovation creation: Institutional constraints in a public sector oriented market. Scandinavian Journal of Information Systems, 23(1), 29–58.
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human–Computer Interaction, 13(4), 373–410. https://doi.org/10.1207/S15327590IJHC1304_03
Hertzum, M. (2021). Organizational implementation: The design in use of information systems. Springer. https://doi.org/10.1007/978-3-031-02232-6
International Organization for Standardization. (2018). Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). https://www.iso.org/standard/63500.html
Krause, P., & De Lusignan, S. (2010). Procuring interoperability at the expense of usability: A case study of UK National Programme for IT assurance process. In B. Blobel, E. Þ. Hvannberg, & V. Gunnarsdóttir (Eds.), Studies in health technology and informatics: Vol. 155. Seamless care – Safe care (pp. 143–149). IOS Press. https://doi.org/10.3233/978-1-60750-563-1-143
Kushniruk, A., Beuscart-Zéphir, M. C., Grzes, A., Borycki, E., Watbled, L., & Kannry, J. (2010). Increasing the safety of healthcare information systems through improved procurement: Toward a framework for selection of safe healthcare systems. Healthcare Quarterly (Toronto, Ont.), 13(special issue), 53–58. https://doi.org/10.12927/hcq.2010.21967
Lauesen, S. (1998). Usability requirements in a tender process. Proceedings 1998 Australasian Computer Human Interaction Conference, OzCHI’98 (Cat. No.98EX234), 114–121. https://doi.org/10.1109/OZCHI.1998.732203
Lee, L., Williams, R., & Sheikh, A. (2016). How does joint procurement affect the design, customisation and usability of a hospital ePrescribing system? Health Informatics Journal, 22(4), 828–838. https://doi.org/10.1177/1460458215592915
Lehtonen, T., Kumpulainen, J., Liukkonen, T. N., & Jokela, T. (2010). To what extent usability truly matters? A study on usability requirements in call-for-tenders of software systems issued by public authorities. Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 719–722. https://doi.org/10.1145/1868914.1869013
Markensten, E., & Artman, H. (2004). Procuring a usable system using unemployed personas. Proceedings of the Third Nordic Conference on Human-Computer Interaction (NordiCHI ’04), 13–22. https://doi.org/10.1145/1028014.1028017
Mehdiyev, N. (2022). Decision making on firm’s software selection with type-2 fuzzy data. In R. A. Aliev, J. Kacprzyk, W. Pedrycz, M. Jamshidi, M. Babanli, & F. M. Sadikoglu (Eds.), Lecture Notes in Networks and Systems: Vol. 362. 11th International Conference on Theory and application of soft computing, computing with words and perceptions and artificial intelligence—ICSCCW-2021 (pp. 480–487). Springer. https://doi.org/10.1007/978-3-030-92127-9_65
Mirel, B. (2004). Interaction design for complex problem solving: Developing useful and usable software. Morgan Kaufmann.
Moe, C., Newman, M., & Sein, M. (2017). The public procurement of information systems: Dialectics in requirements specification [Diagram]. European Journal of Information Systems, 26, 143–163. https://doi.org/10.1057/s41303-017-0035-4
Moe, C. E., & Päivärinta, T. (2011). Challenges in information systems procurement in the Norwegian public sector. In M. Janssen, H. J. Scholl, M. A. Wimmer, & Y. Tan (Eds.), Lecture notes in computer science: Vol. 6846. Electronic government (pp. 404–417). Springer. https://doi.org/10.1007/978-3-642-22878-0_34
Nielsen, J. (1994). Usability Engineering. Morgan Kaufmann Publishers Inc.
Novack, R. A., & Simco, S. W. (1991). The Industrial Procurement Process: A Supply Chain Perspective. Journal of Business Logistics, 12(1), 145.
Pan, Y., Komandur, S., & Finken, S. (2015). Complex systems, cooperative work, and usability. Journal of User Experience, 10(3), 100–112. https://uxpajournal.org/complex-systems-cooperative-work-and-usability/
Po-An Hsieh, J. J., & Wang, W. (2007). Explaining employees’ extended use of complex information systems. European Journal of Information Systems, 16(3), 216–227. https://doi.org/10.1057/palgrave.ejis.3000663
Redish, J. (2007). Expanding usability testing to evaluate complex systems. Journal of User Experience, 2(3), 102–111. https://uxpajournal.org/expanding-usability-testing-to-evaluate-complex-systems/
Riihiaho, S., Nieminen, M., Westman, S., Addams-Moring, R., & Katainen, J. (2015). Procuring usability: Experiences of usability testing in tender evaluation. In H. Oinas-Kukkonen, N. Iivari, K. Kuutti, A. Öörni, & M. Rajanen (Eds.), Lecture notes in business information processing: Vol. 223. Nordic contributions in IS Research (pp. 108–120). Springer. https://doi.org/10.1007/978-3-319-21783-3_8
Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14, 131–164. https://doi.org/10.1007/s10664-008-9102-8
Savioja, P. (2014). Evaluating systems usability in complex work—Development of a systemic usability concept to benefit control room design [Doctoral dissertation, Aalto University]. Aaltodoc publication archive. https://urn.fi/URN:ISBN:978-951-38-8146-7
Savioja, P., & Norros, L. (2013). Systems usability framework for evaluating tools in safety–critical work. Cognition, Technology & Work, 15(3), 255–275. https://doi.org/10.1007/s10111-012-0224-9
Skjott Linneberg, M., & Korsgaard, S. (2019). Coding qualitative data: A synthesis guiding the novice. Qualitative Research Journal, 19(3), 259–270. https://doi.org/10.1108/QRJ-12-2018-0012
Still, B. (2011). Mapping Usability: An Ecological Framework for Analyzing User Experience. In M. Albers & B. Still (Eds.), Usability of Complex Information Systems: Evaluation of User Interaction (pp. 89–107). CRC Press.
Strand, I., Ramada, P., & Canton, E. (2011). Public procurement in Europe. Directorate-General for the Internal Market and Services (European Commission). https://op.europa.eu/s/xeEA
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Turner, J. R., & Baker, R. M. (2019). Complexity theory: An overview with potential applications for the social sciences. Systems, 7(1), Article 4. https://doi.org/10.3390/systems7010004
Tyllinen, M. (2023). Process for usability evaluation for selecting an information system in public procurement—Measuring usability of health and social welfare IS [Doctoral dissertation, Aalto University]. Aaltodoc publication archive. https://urn.fi/URN:ISBN:978-952-64-1344-0
Tyllinen, M., Kaipio, J., Lääveri, T., & Nieminen, M. H. T. (2016). We need numbers! Heuristic Evaluation during Demonstrations (HED) for measuring usability in IT system procurement. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 4129–4141. https://doi.org/10.1145/2858036.2858570
Zou, W., Brax, S. A., & Rajala, R. (2018). Complexity in product-service systems: Review and framework. Procedia CIRP, 73, 3–8. https://doi.org/10.1016/j.procir.2018.03.319
