A Critical, Creative UX Community: CLUF

Research and practice in human-computer interaction (HCI) are now in their fourth decade. After a first decade focused on modeling and guidance, professional practices associated with HCI have focused first on user-centered design (UCD), next on user experience (UX, which includes usability), and now on interaction design (IxD), with a range of practices (e.g., creative, agile, lean) that challenge established ways of working within UX, UCD, and early HCI. In this editorial, I advocate a new form of interactive community publication, complementary to JUS, to respond to new creative emphases within human-focused interaction design practices and research. I have called this CLUF (creatively led user foci), pronounced like the Northern English word clough, meaning a steep valley or ravine. The realities of reflective creative practices are that we can always probe further and explore more as we work down through layers of design practice. CLUF would support a much needed online community of practice around systematic rigorous exploration of creative UX.

User Foci in Interaction Design: (My) Past, Present, and Future

We can roughly divide the four decades of HCI into basic research, applied research, maturing practice, and balanced practice. Basic research in 1980s HCI supported modelling and design guidance. Applied research in 1990s UCD attempted to compare and validate new discount methods. Maturing practice in 2000s UX cut free from academic research and innovated within the practitioner community. Balanced IxD practices in the 2010s must integrate user foci alongside creative, commercial, and technical practices. These shifts require a new form of community publication that is better able—than JUS—to support creative UX practices within IxD. I support my position in this editorial by drawing on personal experience within HCI research and practice since 1981.

My career has been diverse since graduating with a joint degree in History and Education. As a high school teacher of history and social studies, I became interested in using computers in the classroom and developing computer games. This led to a PhD in Computer Science within an HCI research group, and 25 years of research and practice in the design and development of interactive software and media. Over the last decade I have moved from the design end of computing to the computing end of design, and more recently into communication and media. Together, I have a broad range of perspectives that I employ in this editorial to the present and past of user foci in IxD, by which I mean any design or development practices in software or media that focus on users or usage. There are of course other equally important foci for design and development: A key challenge for HCI, UCD, and UX has been their ability to co-exist effectively with these other foci.

As someone originally educated as a historian, I bring the past into the present as a basis for understanding where we are and where we need to go. I will argue that UX needs to evolve to support creatively led user foci, and that JUS should establish an on-line community, called CLUF, around case studies and practitioner resources. In building this argument, I will draw on the history of user-focused practices in the development of interactive software and media.

Where We Have Been and Where We Are Now

Gould and Lewis (1985) promoted three key principles of usability: early focus on users and tasks, empirical measurement, and iterative design. The implication of these principles were not universally recognized or acted upon in 1980s HCI and software development. Instead, research and practice tended to be design led, with models of applications being required as a pre-requisite for modelling tasks. The modelling and analysis of the 1980s generally focused on the tasks that resulted from an interface design, rather than being primary data inputs to IxD. It was thus not possible to analyze tasks until a design existed in some form, for example, as a model, specification, or prototype. However, until a design exists in some form, there can only be an early focus on users and their goals because tasks can only be considered in terms of the initial states from which users must reach goals. Unlike user goals, initial states and task methods are design dependent, as they can vary across alternative designs. In this sense, much of the 1980s HCI was not user centered.

For my first decade of HCI practice, there were no published methods for contextual research or user testing, and thus limited support for putting Gould and Lewis’ principles into practice. My practice was thus closer to that of a creative designer than an evidence-based UCD specialist. However, software engineering values tended to promote specification, modeling, and analysis over creative design, and thus the realities of HCI practice in the 1980s were masked by publication practices that emphasized rational analysis over creative practice. It is only over the last decade that the need for creative practices within UX work has come to be overtly recognized. Before this, UCD research and practice focused on methods that introduced systematic user foci for the first time to interactive software.

The 1990s UCD decade began with a stream of “discount” usability methods, ending with ones for contextual design (e.g., Beyer & Holtzblatt, 1998). The former originated in academic or industrial research (Cockton, Lavery, & Woolrych, 2003), but the latter came from practice. These discount methods promised re-usable practices for contextual research and usability evaluation as practical bases for Gould and Lewis’ principles. What were discounted in these UCD methods were human science practices for UCD work. However, this was not matched when attempting to compare and validate discount methods, where human science standards persisted. Looking back, how could discounted (i.e., less rigorous) methods ever pass rigorous validation tests? It should thus have been no surprise when a seminal study exposed severe problems with the research quality of assessments of UCD methods (Gray & Salzman, 1998). Next, problems were exposed with UCD methods themselves, especially evaluator effects (Hertzum & Jacobsen, 2001; Molich. n. d.). New research approaches were developed to improve the validity of evaluation method assessment, but their main impact was to confirm evaluator effects as inescapable (Cockton & Woolrych, 2001), but still reducible (Cockton, Woolrych & Hindmarch, 2004).

The modelling of 1980s HCI has thus been succeeded by the muddling of 1990s UCD. Both decades had been dominated by academic research. The seeds for UX mastery by practitioners in the 2000s had been sown in 1991 with the establishment of the Usability Professionals Association (UPA, now UXPA). Following the publication of Contextual Design: Defining Customer-Centered Systems (Beyer & Holzblatt, 1998), a book for practitioners by practitioners, all significant developments in UX approaches originated and developed within practice and not academic research. For example, personas (archetypes that represent multiple users) were first outlined by Alan Cooper (1999), but several years later were given a systematic basis resulting from a community effort led and disseminated by Jonathan Pruitt and Tamara Adlin (2006). However, there was also a substantial external driver in the 2000s after the dotcom crash. Hype, promises, and dreams had to give way to reliable high standards of UX work. The profession responded appropriately.

Practitioner authored books and articles benefited from advantages over academic papers presented at scientific conferences and in journals, most of all because the UX community began to see the value of the information based on professional expertise and experience, as evidenced by practical examples, over academic credentials and practices. The initiative in UX practice had shifted from academically focused UCD in the 1990s, and few contributions were made from academic research. It became much more difficult to disseminate new UCD methods in refereed publications after the 1990s. There were exceptions, such as the journal article “Metaphors of Human Thinking for Usability Inspection and Design” (Frøkjær & Hornbæk, 2008), but continuing expectations for scientific validation presented major challenges that could only be met by exceptional research teams. Similarly, no paper on personas was accepted at ACM CHI conferences until McGinn and Kotamraju (2008).

Academic research on UCD and UX changed direction during the 2000s decade of practitioner mastery. My first two PhD students had been early pioneers (both went on to work at Microsoft): Darryn Lavery was interested in evaluation methods; Steven Clarke was interested in contextual design. Both had computing degrees and had learned IxD largely as I had, in design-led software engineering contexts. However, their research heralded a new form of study that directly involved software designers and evaluators in understanding methods in use. This resulted in tools for both practitioners and researchers. Steven Clarke developed a tool to link contextual research data and models to design specifications and analyses (Cockton & Clarke, 1999). Darryn Lavery developed resources that could be used for both evaluation research and practice (e.g., structured report formats, problem extraction procedures; Cockton & Lavery, 1999). These techniques developed by my former students refocused my research on knowledge resources for UCD as a more realistic and fruitful focus than discount methods.

The unique position of my group at Sunderland University (1997–2009) made much of this new focus possible. We combined HCI teaching and research with extensive commercial and publicly subsidized pro bono practice. In 1999, I became director of a regional support project for digital media companies (followed by a second regional project in 2003–2005 and a final national one from 2003–2007). Within this venue, my researchers offered usability consultancy work, informed by an increasingly sophisticated understandings of how usability evaluation actually worked in practice. This work was always successful; sometimes exceptionally so, with over one million UK pounds worth and also 35% rises in website sales following re-design work. However, the usability work was only one part of each project—with creative, commercial, marketing, and other strategic inputs from a range of collaborating professionals. The usability work was not all empirical, but instead used inspection practices that I and others had been developing since the 1980s.

From 2004 to 2009, we were able to further develop perspectives on evaluation resources in collaboration with the European Cooperation in Science and Technology (COST). In this timespan, researchers and practitioners from over 25 countries collaboratively worked towards MAturation of information technology USability Evaluation (MAUSE) within a flexible and efficient networking forum known as a COST Action (Cockton, Woolrych, Hornbæk, & Frøkjær, 2012; Woolrych, Hornbæk, Frøkjær, & Cockton, 2011). We found that our experience of the importance of unique project practices due to unique local resources was shared by dozens of European usability researchers and practitioners. We had been charged with finding ways to improve comparisons of usability evaluation methods, but concluded instead that complete methods only come into being through project work. What pre-exists project work are loose incomplete collections of resources (often partial or even minimal) that we called approaches in contrast to methods (Woolrych et al., 2011). We saw many examples of creative practice in UX work, and we viewed this as an indispensable aspect of the UX mastery of the 2000s.

Practitioner mastery in 2000s UX books was based in expert combination and adaptation of a wide range of public and local resources. However, UX masters rarely presented their work as creative contingent practices. Despite academic and professional research that showed the limits of re-usable methods as a basis for practice, UX masters could still view methods as procedures that could be described and explained to readers. As a result, UX masters did not see themselves as inimitable maestros. Instead, they regarded their new methods as repeatable practices that could be communicated to UX apprentices through detailed examples. The result was very lengthy books that proved to be too detailed for widespread diffusion of expertise. Interestingly, shorter versions of the original publications on both contextual design (Holtzblatt, Wendell, & Wood, 2005) and personas (Adlin & Pruitt, 2010) were published in the 2000s.

Mastery, it seems, could not be completely commoditized and had to be pruned back to rapid essentials. Implicitly, these UCD-lite texts signaled a move away from methods as commodities. As 1990s muddling gave way to 2000s mastery, dissemination still shipped out the method—the whole method and nothing but the method. However there is more to mastery than method. To communicate the whole method, and nothing but the method, is to devalue and diminish the hard work required to get methods to work. As with all human performance, there is a wide range of attainment. Mastery cannot be instantly, or even ever, understood or attained by all. We are now almost half way into the 2010s. After decades of first modelling, then muddling, and next mastering, where are we headed now? I find myself looking to my move from computing to design, where maestros are the norm. We expect exceptional designers. We educate and develop design students as individuals who will never graduate with similar, never mind identical, profiles. We do this not because we are elitist, or deluded romantics, or lazy, or lacking in objective knowledge. On the contrary, we do this because of a millennia (e.g., Vitruvius De Architectura 80 A.D.) of objective evidence make it clear that design excellence has never rested upon, or has even been supported in any way by, validated methods with rigorously assessed performance. While HCI modelled, UCD muddled, and UX mastered, creative design has been at ease with maestros because there simply is no alternative. Any honest look at today’s IxD landscape would lead to a similar conclusion: We have gurus and celebrity maestros who are celebrated for their achievements and influence, and not for the p values of their method validations.

User foci in IxD have thus moved from modelling via muddling and mastery to maestro-ing. The rise of gurus and celebrities is marked by a reduction in ambitions for research and practice. There is evidence of UX maestros adjusting to the challenges of method validation and communication alongside those of creative practices. Keynote presentations, master classes, and promotional blog posts are best suited to a focus on resources below the level of methods, or on design options. My current unread Alertbox articles from Jakob Nielsen thus include short pieces on interface animation and image size, and short training video demonstrations of think aloud. Only the latter is a UX resource; the other two are IxD resources in support of effective creative design.

My sense is that UX celebrity is increasing while the topics that celebrities focus on are shrinking. The two may well be related. Credibility (having a following and an impact) is easier to maintain through focused actionable communication, whether on stage or in a blog. Modelling, validation, comparison, and method mastery are all intellectually fraught with issues, unlike expert craft knowledge and creative practices. Significant challenges of communicating and coaching can become manageable with a sufficiently tight focus on well-evidenced and effective practices.

Despite the growing number of UX celebrities, creative practices are not a recent development. I first personally became aware of changing requirements for UX work in 2003 (interestingly, the year when IxDA was formed) from a lunch conversation with Giles Colborne (UK UPA President 2003–2007) who I’d asked about requirements for new graduate hires in UX. To my surprise, he said creative design skills were highly advantageous. HCI graduates who could mock up and prototype user interfaces were increasingly sought after in the UX job market. This trend was reinforced in Stephanie Rosenbaum’s (2008) chapter for the MAUSE project book (Law, Hvannberg, & Cockton, 2008), where she described the addition of visual prototyping in the 2000s to TecEd’s consultancy offer. Such expansions in usability practice meant that user-centered work was no longer enough: consultants also had to be able to center on design. JUS and the UXPA need to accept and embrace these trends, and develop new formats that are suitable for publication, discussion, critique, and dissemination of creative UX work.

So, for me, that’s where we are, and that’s where we’re going to stay: Maestros aren’t going to go away. The historian in me knows that where we are carries with it everywhere that we have been. Disciplines are like geologies, with a succession of strata on top of each other, often distorted and bent, with older rocks sometimes revealed again as younger ones wear away. We cannot escape the past and the legacies of HCI modelling, UCD muddling, and UX mastering, while we shape the realities of emerging IxD maestro-ing. There are implications here for JUS: some can be accommodated with current values and practices; others require a complementary interactive publication to respond to the realities of creative UX practices.

What Do Masters and Maestros Mean for JUS?

JUS was founded in 2005 in the mastery decade of UX practices, at least partially as a response to the difficulty of publishing double-blind reviewed research on UCD methods, as the dust settled on the muddling decade. The fortunes of JUS depend on whether its reviewers prefer mastery in UX practice over muddling in UCD research. Some (perhaps many?) JUS reviewers remain attached to the standards from the muddling phase of UCD research, with expectations for controlled replicable studies that are impossible when studying work that must rely on professional judgment and expertise. In this respect, the results of an analysis of JUS papers is revealing (Dumas & Saparova, 2012, p. 6):

About half of the papers are categorized as Case Studies by the authors and 14% are experiments. When we read the papers, we categorized many more of them, 40%, as experiments. Apparently, authors are reluctant to use the term “experiment” to describe the study they report. We can only speculate, but perhaps they feel that reviewers will treat their paper more harshly if they call a study an experiment.

Whatever the true extent and rigidity of scientific standards in JUS reviewing, there may be evidence here of authors’ concerns about inappropriate reviewing standards. Categorizing a paper submission as a “case study” indicates a claim for project specific outcomes that should have significance for UX professionals who have the judgment and expertise required to relate it to their practice. Categorizing it as an experiment indicates a claim for objective generalizable knowledge that must be judged by the highest scientific standards. However, what is clear from the muddling decade of 1990s UCD research is that scientific validations and assessments of methods are impossible, and evaluator and researcher effects are unavoidable. This does not mean that scientific studies of UCD or UX work are impossible, but this is more straightforward when studying a single method in use (e.g., Cockton & Woolrych, 2001) or comparing the impact of specific resources such as short/detailed task specifications on cognitive walkthrough (Sears & Hess, 1999), extended structured report formats on usability inspection performance (Cockton, Woolrych, & Hindmarch, 2004), or realistic/stock photos on persona credibility (Wisser & Stappers, 2007).

Appropriate support for case studies on UX practices, supported by guidance and advice for authors as well as appropriate reviewing values and practices, should allow JUS to continue to serve UX researchers and professionals. However, it is not clear how the predominantly scientific values associated with JUS, however pragmatically these are interpreted and applied, can make space for more creative UX within IxD. In these circumstances, a new complementary venue may be required. I propose that this is the case. Before elaborating on this, I review how I came to understand creative design practices, and summarize what I understand these to be.

Creative Design Practices

In the many success stories of Gould and colleagues (Cockton, 2008b), design was a black box into which they unwittingly shed light to reveal many innovative creative design-led activities at the heart of their own self-proclaimed human-centered practices. As human scientists, this is understandable, as they would have preferred Herb Simon’s rational evidenced The Sciences of the Artificial (1981) to Donald Schön’s personal The Reflective Practitioner: How Professionals Think in Action (1983). But despite orthodox normative accounts of scientific work, the reality lies closer to creative practice than to idealized rigor. A key question for UX work in support of IxD is whether it will cling to the values of its human science and engineering UCD progenitors. I must admit that it took me all of the 2000s to realize just how much change was needed to achieve effective interworking between UX and creatively-led IxD.

Over the last few years I have come to realize how much UCD research on methods had overlooked well-established results from design research and in science and technology studies. In the former, studies showed that normative accounts of rational design (as advocated, for example, by Herb Simon, 1981) did not reflect realities as shown by Bucciarelli (1994) for engineering design, and Schön (1983) for creative and/or professional work. Similarly, Gould and colleagues’ advocacy of their three key principles of usability had little basis in their own evidence, which provided no examples of any use of empirical measurement (Cockton, 2008b), and thus no evidence of its worth. What their accounts did do, however, was reinforce my consultancy and research experiences. Their success stories all involved long term user involvement in design-led projects with deployed prototypes and operational systems. Pre-deployment testing made extensive use of informal practices such as hallway testing, just as I had in my very first IxD work as a school teacher.

It is easy to work alongside creative designers for short periods of time and erroneously conclude that we understand creative design as a result. In 2005, I was very fortunate to be awarded a UK NESTA Fellowship to study and develop value-centered approaches to IxD. As well as being able to buy myself out for 40% of my time for almost 2.5 years and having a travel and materials budget to give me the freedom to study, practice, and visit, I was required to have a mentor. My fortune grew even more when Gillian Crampton-Smith agreed to be my mentor, and more again when her husband Philip Tabor took an interest in my fellowship. At this point, I had been an interaction designer and then an HCI researcher for almost 25 years, but I had never worked with design practitioner academics. In fact, my contact with creatively trained designers had been very limited: working with a fine artist to create icons and schematics for Nortel digital-switch maintenance user interfaces (1988-89) and working with a graphic designer to create icons and layouts for the redesign of a business information system that was included in an international design exhibition (1996). Incidentally, the graphic designer invented ribbons (double-height toolbars, but without tabs) in this project, breaking Windows 95 toolbar standards, but clearly strong both visually and usably. They neither originated from, nor were validated by, user research. This creative response to an aesthetic problem demonstrates the value of design-led approaches.

Gillian and Philip steadily introduced me to what has come to be known as design thinking, but perversely is very poorly communicated by most explicit accounts of it (a notable exception to this is Nigel Cross’ masterly Design Thinking, 2011). Their respective backgrounds in visual design and architecture exposed me to the thinking and practices that Schön had foregrounded in his tacit challenge to Simon’s The Sciences of the Artificial. However, I largely continued to persevere with approaches that Simon may well have endorsed and recognized as rationalist design practices, although within the context of a creative design focus on value (as worth). I thus sought to better focus existing user-centered evaluation practices on achieved worth (Cockton, 2007; Cockton, 2008c), and through a critique of Gould and Lewis’ three principles of designing for usability (Cockton 2008b), I sought principles that could span all design activities, not just user-focused ones.

Such engineering design perspectives were compatible with my status as a Professor in a Computing Department, but that changed after my NESTA fellowship ended in 2008. Rubbing shoulders with creative designers had not been enough for me to develop an accurate understanding of creative design. Although, I was finally becoming aware of some of the key realities of creative design, two decades of work within an engineering design paradigm made it difficult for me to immediately see the implications of creative design for my existing IxD practices, including development of new approaches such as worth maps (Cockton, 2008a; Cockton, Kirk, Sellen, & Banks, 2009; Cockton, Kujala, Nurkka, & Hölttä, 2009), which continued the focus on modelling and specification of 1980s HCI. In this sense, my eyes were as closed as Gould and colleagues to the realities of their project teams’ own creative design practices that depended on design practices that UCD often aimed to eliminate.

In 2009, I moved to Northumbria University’s School of Design, with its roll call of outstanding alumni, including Sir Jonathan Ive (Apple) and Tim Brown (IDEO). Once I had been permanently embedded in a creative design context for a few years, I finally began to understand how different it was to engineering design practices such as usage-centered design (Constantine & Lockwood, 1999). Now, engineering design practices are creative, but creativity is restricted to the “fuzzy front end of design” and severely constrained after problem and requirements specification. The use of the words “creativity” and “creative” is instructive. Creativity is a practice that can be introduced and excluded at different stages of engineering design work (including software engineering). Being creative is pervasive, and thus few creative design practitioners refer to creativity as some optional add on.

At Northumbria, I first worked alongside a team of contract researchers who had been partners in my second regional support project for digital media companies (CODEWORKS NITRO 2003–2005). Through those researchers, I came into contact with some seminal papers in design research, albeit not in the chronological order below, and these finally drove home what I had been exposed to by Gillian and Philip on my NESTA fellowship.

Darke (1979) is a seminal paper on design practice. In a study of architects in urban planning, she found that they would first fix on an important aspect of the problem, develop a crude design (a primary generator) on this basis, and then examine it to see what else they could discover about the problem. In contrast to the analysis-synthesis model of engineering design, with a Chinese Wall of requirements between the two, problem and solution spaces co-evolved (Dorst & Cross, 2001), with elements of solutions emerging very early in the design process. This was in strong contrast to the position of Gould and Lewis (1985) where no design activities were allowed to start until users and tasks were properly understood. From a design point of view, this is simply nonsense, because without some idea of what the solution may be, it is impossible to focus on any users or tasks: users of what? tasks for what? In the absence of some design vision or brief, there are only people and their existing activities. None are potential users yet of anything who could become engaged in any possible tasks.

Davies and Talbot (1987) interviewed elite UK designers (Royal Designers for Industry—RDIs) and noted the prevalence of an imago, which was similar to Darke’s primary generator. An imago represents the current sense of a preferred solution. Such an imago was arrived at intuitively, with the reasons or the process origins for it hard to articulate rationally, but at the same time the RDIs claimed to know when an idea was the right one, and associated this with deeply felt positive, ecstatic pleasure and a lack of anguish. This replicated 1960s studies of American architects: Problem and solution were seen as a unique complete whole. This wholeness resulted from openness to all kinds of experience, and was affirmed emotionally not rationally.

The inseparability of problem and solution spaces and frames is a core characteristic of wicked problems, as most recently formalized by Conklin (2006), where a problem cannot be fully understood until a solution has been committed to and adequately refined. There is no adequate stopping rule for the latter, and no basis for deciding whether a solution is right or wrong. Every wicked problem is novel, unique, and without given alternative solutions. Every solution to a wicked problem is a one-off, and little if anything may generalize to future designs. Given this, a designer’s confidence in a primary generator or imago is indispensable. Without one, nothing can move. No problem specification will appear from nowhere, nor can any requirements be expressed in a form that can be verified. In the absence of given definitive requirements, the designer’s ambitions and values are crucial. Cross (2011) quotes the architect Sir Denys Louis Lasdun:

Our job is to give the client, on time and on cost, not what [s]he wants, but what [s]he never dreamed [s]he wanted; and when [s]he gets it, [s]he recognizes it as something [s]he wanted all the time.

At the heart of Lasdun’s position is generosity. Creative designers expect to give far more than is expected, or even imagined. They take responsibility for design outcomes. UX professionals and researchers have tended to be more concerned with design inputs, and make the validity of data, analyses, and conclusions a primary concern. However, validity has no automatic value for design. What matters are direction, relevance, and inspiration. Flaws in the validity of design inputs will not automatically translate into flaws of direction, relevance, and inspiration. Designers should ensure that they do not. Design inputs that provide strong direction, relevance, or inspiration, but turn out to be seriously flawed in their implications for design, will reveal themselves as design progresses, allowing corrective action to be taken. In contrast, design inputs that provide no direction, relevance, and inspiration will provide no value at all to design work. Validity has no inherent value, and invalidity has no automatic adverse consequences. The relationships between problem and solution spaces are too complex to allow any simple inevitable causal relationships here. True, this runs counter to scientific values, but it does not run counter to design ones. UX is about design, not science.

These key understandings of the realities of creative design guided me in my previous role as Associate Dean for Research and Innovation in Northumbria University’s School of Design, where one of my key responsibilities was to support practice-based researchers in the development of their research portfolios for national UK assessment. Research elements had to be expressed in terms of the imperatives that drove creative practice, as well as the process through which creatively-led research came to create new knowledge. Support for curating portfolios has been improved by research within the contexts of HCI and IxD (Koskinen et al., 2011) that has systematized the nature of constructive design research, that is, research that is methodologically based on a creative rather than scientific process, but is nevertheless rigorous through practices of recording and reflection.

Creative processes may show evidence of regularities across projects, designers, and communities, but this does not result from following methods. Design methods have been very controversial in design research and practice, but HCI and UX research and practice appear to have been unaware of the extent or longevity of this controversy, which dates back to divisions within the successor to the Bauhaus at Ulm in Germany (these divisions led to Ulm losing its funding in 1968). Many UCD principles are compatible with the “Ulm Model.” Their advocates have tended to have had backgrounds in engineering design where methods that calculate are well established and effective (although also largely automated now within computer-aided design). Such methods can be regarded as instruments that can be applied consistently and effectively across design projects. However, this is not the only possible way to conceive of methods. Keinonen (2009) contrasted instruments with competences and agendas. Competences are one form of resource that are required to put practical methods in place from a starting point of loose incomplete approaches (Woolrych et al., 2011). Effective method use is not possible without the required competences and other resources being present. Given this, evaluator effects are unavoidable. Agendas are even less well defined than competences. UCD is an agenda, and this is what Gould and Lewis (1985) actually articulated, rather than a process or the methods underpinning it.

UX and IxD work are far better understood as a creative interaction of competences and agendas than as a set of validated repeatable methods (without evaluator or researcher effects) that can be applied “as is” across a range of project contexts. The ideals of engineering design, and its underlying scientific agendas, are not a realistic basis for much UX work. Far from being at odds with creative practices that must be restrained by evidence and reason, UX work is a set of creative practices. There are objective aspects within it, and there are activities that can be effectively planned, but as a whole, UX processes are unavoidably dynamic and creative. As such, UX research and practice need to look to creative design, as well as engineering design, for its values and practices.

In looking to creative design, UX research and practice must be able to accommodate two groups of fundamentals of creatively led design (research):

1. There are limits to planning and control.
   • No creative activity can be fully pre-planned.
   • There can be no preset order to creative design activities.
   • Creative activity cannot be controlled, but it can be paused and reflected on.
   • Reflective practices guard against neglect and oversight.
   • Existing research practices may require adaptation, even at the expense of validity.
2. Bias is not a problem, but poor design is.
   • Designers design, and so must UX and IxD specialists.
   • Designers differ, and so do UX and IxD specialists.
   • Design outcomes are only partially the result of design inputs: the best result from the generosity of designers (including UX and IxD specialists).
   • Poor quality of primary data need not translate into poor quality of design outputs.
   • Poor quality of UX expertise and IxD practice will translate into poor quality of design outputs.

To accommodate these two groups of fundamentals, UX work within creative design and technology contexts requires maestros, not just masters. Maestros can confidently and creatively apply their professional judgment and expertise in ways that are highly contingent on project contexts. Confidence in judgment is inherently subjective. It does not involve appeal to evidence in isolation. It does not involve objective truths. Instead it involves professional recommendations based on a holistic understanding, as evidenced subjectively by a level of confidence that approaches that of an elite designer. Most importantly, by understanding and adopting the practices of creative design, all UX work can become compatible with it.

Creative practices can make UX compatible with contemporary agile and lean development practices (Gothelf & Seiden, 2013). Agile practices are design led, not user research led. Lean value systems associated with agile software development practices cannot assume that any UCD activity has inherent value, as Gould and Lewis (1985) did in their three key principles for usability. Instead, UX practices have to be shown in the moment to deliver value at the current point within a project. This is much more possible in creative practices where problem and solution spaces co-evolve, allowing user research to focus on currently identified assumptions about a design, and potentially reframing the design problem as a result. However, without a design solution with some degree of advancement, there can be no (implicit) assumptions to explore through user research. User research here can be either contextual or evaluate usage. Either way, the value delivered can only be understood relative to a current potential design solution. In this sense, IxD must always be design led and cannot be continuously focused on users. Contextual user research that is motivated by the implications of a design is not strictly user-centered, but instead focuses temporarily on users in an attempt to validate current design assumptions. If design can be centered on anything, it can only be centered on itself. Its center cannot lie outside.

I would be very surprised if most UX professionals cannot already recognize elements of their own practice above. Although I did not have a good understanding of creative design values and practices a decade ago, my group at Sunderland nevertheless developed new forms of agile creative UX practices. For example, on one project with a generous (five digit) user testing budget, we took advantage of a large test user pool to change the planned (and signed off) task set twice (with client permission each time). We reported usability problems and bugs on a daily basis by an email to the system developer, who sent updated software each day. Within two weeks, we ended up with a stable system with no known bugs or usability problems, so we used the third week for out of the box and manual development and testing. This was in 2004, long before anyone had proposed Agile or Lean UX practices (Gothelf & Seiden, 2013). Evaluation video was used to persuade joint venture partners, extending the value of UX work beyond design changes.

Such practical successes provided evidence that was still largely lacking in the literature on how usability studies can contribute to better digital products and services (and happy clients too), but this was always in the context of the creatively, strategically, and technically led redesign work that actually delivered the benefits. Close agile effective collaborations here were much eased by understandings from our research that usability work was unpredictable and creative, and involved considerable professional judgment and expertise. Textbook methods had little, if any, role in successful follow through from usability work. We did not work to a scientifically rigorous study plan, but instead collaborated within flexible and agile multi-disciplinary teams. We did not lose sleep over scientific rigor, but would have lost sleep over waste or uncooperative behaviors resulting from human science value systems, and any resulting client unhappiness.

More recently, I have collaborated with two colleagues to analyze an innovative approach, “Creative Sprints: An Unplanned Broad Agile Evaluation and Redesign Process,” that resulted within a rapidly evolving UX consultancy context (Garnik, Sikorski, & Cockton, 2014). Analysis has revealed that this work shares all the core features of creative design. Firstly, the problem and solution co-evolved through the following problem solution pairs:

• finding usability problems and redesigning a user interface for a call-center module
• finding user experience issues and redesigning internal services (e.g., IT support)
• finding customer experience issues and redesigning service value chains

This creative setting resulted from the consultants (Garnik and Sikorski) being unable to operate the call-center module for a usability inspection due to customer privacy issues. Instead, three experienced senior operators demonstrated tasks to expose usability problems that had been identified before the consultants arrived through email crowdsourcing. The problem owner (an IT manager) was also present in the demonstrations. After each task was demonstrated, sometimes further usability problems were revealed, with discussions adding insights on problems with operator UX and customer experience (CX). Causal analysis identified factors beyond the target module’s user interface. Brainstorming on possible solutions was supported by advice on feasibility from the IT manager and on management priorities and preferences from the senior operators. The consultants came to see each cycle of demonstration, discussion, analysis, and change recommendation as a “creative sprint.”

The first core feature shared with creative design was thus that the problem and solution spaces co-evolved over a few dozen creative sprints. The second was that reflection between sprints let the consultants form explicit understandings of, and commit to, this changing scope, which also marks their work as a creative activity. Thirdly, their willingness to move beyond usability inspection via operator UX and CX to redesigning service value chains is a further creative practice of generosity. Whereas engineering design delivers to specification, creative design always delivers beyond it, as evidenced by the Cross (2011) quote from Lasdun above.

Fourthly, as creative UX activities, creative sprints are not fully plannable, predictable, or replicable. The consultants were able to reframe their practice as agile and to allow the problem and solution space to co-evolve due to their joint knowledge of agile development, experience design, and service design. Their design knowledge differed, and with the local call-center team they formed a new evaluation and re-design approach on the fly. The outcomes of this work were more due to the combined generosity of the consultants and the local team than to any formal evaluation inputs (there was only the crowd sourced set of usability problems, grouped and ordered by the senior operators—there was no primary data from any contextual research). Without the combined expertise of the consultants and local team, who soon became a single collaborative team, the work would not have resulted in such a high quality set of comprehensive design change recommendations ranging from user interface changes for one module to changes to internal services and policies and a review of current service value chains.

Carrying out usability work in commercial contexts, both in the example above and in my group at Sunderland, foregrounds product and service goals as key determinants of severity for usability problems. Usability problems cannot be objectively or universally defined, nor could universal severity scales be devised or used reliably. Instead, a usability problem with any significant consequences was one that reduced the ability of users and/or other stakeholders to benefit from the intended value for a digital product or service. However, I did not originally derive this insight or position from usability work, but instead encountered it when designing services for the regional support project for digital media companies. A business member of the advisory board introduced us to the very basic concept of value propositions.

It was through designing the services (including usability consultancy) that I originally recognized the inherently subjective nature of design and evaluation work. Design seeks to create value. Evaluation should aim to measure the achievement of this value. While performance can be objectively measured, a position on, for example, how fast is fast enough or how slow is too slow, is inherently subjective. This concerns values, which are not facts. Facts can be objective. Values cannot.

A human scientist must strive for objectivity, which is held to require the avoidance of values. In reality, scientific practices are located in work cultures that, as with any culture, are grounded in a system of interlocking values. There can be no scientific basis for these values. Instead, values are the basis for science. Science values are not, however, the basis for design or commercial success, where we encountered values underlying the drive to create, deliver, and communicate value. Note how the English language can confuse here, with two very different senses of value in the singular and values in the plural. To reduce this confusion, I came to stress worth instead of value (Cockton 2006). Worth is the perceived or achieved balance of benefits over costs. Creative IxD, as with all design, must focus on worth as its basis for generosity, alongside reflective co-evolution of problem and solution spaces. The same is true of creative UX, which raises the question of how JUS and UXPA can support the publication, discussion, critique, and dissemination of worthwhile creative UX work. One possible response is presented in the next section.

Staring Into a Clough: The Need for a New Venue for Creative UX Work

JUS can embrace some, but not all, aspects of creative UX work in its current format. Much can be achieved through appropriate support for case studies on UX practices, supported by guidance and advice for authors, and appropriate reviewing values and practices. However, it is not clear how the largely scientific values associated with JUS can accommodate the creative practices that are becoming a major part of contemporary UX work. A new complementary venue is thus needed.

This new venue needs to support both dissemination and Web 2.0 discussion of creatively led user foci (CLUF) in IxD practice. The acronym CLUF is pronounced like the Northern English word clough, meaning a steep valley or ravine. That’s apt, because it is not yet fully clear what is required to support a community of creative UX practitioners. We may all thus feel vertigo as we look down into a dark chasm. Equally, those raised near hills and cloughs, as I have been in Northern England, may feel an urge to scramble down and explore what lies below. Regardless of our birthplaces, I would hope that there is a large enough group of researchers and professionals with a commitment to creative UX work to bring CLUF into being.

As a first step towards creating a CLUF community, I propose the following community rules:

• Case studies of creative and innovative practices must be rigorously documented (using appropriate media, with a structure similar to design portfolios or annotated sketch books) and self-assessed through a process of expert reflection. Contributors to CLUF need to prepare for rigorous defense of their knowledge claims with appropriate documentation and record keeping. The lab books of science become the workbooks of design (Gaver, 2011).
• A supportive reviewing process (initially a small closed group) must be extended beyond publication into open collaborative commentary that can even better identify the relevant existing public body of knowledge as well as discuss the nature of apt challenges. Practitioner researchers should especially benefit from a supportive reviewing process and subsequent open collaborative commentary.
• Knowledge needs to be original, significant, and a rigorously defended contribution to an existing public body of knowledge, which can create problems for practitioner researchers because academic researchers are typically better able to identify the relevant existing public body of knowledge. The significance of original knowledge is demonstrated through critical judgment based on knowledge of relevant theories and achievements.
• Clear knowledge of outcomes that are defended through evidence and argument and are open to apt challenges must be present (i.e., not uncritical impositions of invalid orthodoxies from scientism—Sorell, 1991). Moderation should flag inept postures from scientism.
• JUS should support CLUF by inviting and publishing commentaries on both the state of the art (existing public body of knowledge) and also on standards for reviews and discussions (apt challenges).
• JUS should also support CLUF by publishing rigorous reflection on one or more CLUF case studies, allowing practice-led research to be archived in a recognized journal format.

The outcome of creative design research is both a presented solution and its associated finalized problem framing, and also an assessment of the extent to which the solution solves the problem. The resulting problems need not correspond to existing understandings within the professional community, indeed the best CLUF research should always reframe current understandings of current problems in ways that open up extensive possibilities for highly innovative solutions, which, paraphrasing and extrapolating the previous quote from Lasdun: offer UX professionals not what they want, but what they never dreamed they wanted; and when they get it, they recognize it as something they wanted all the time.

There is to be no requirement for a proof of an existing need, or of limitations of existing solutions and practices (Cockton 2013). Instead, what is required is both better understandings of the challenges facing UX professionals and better ways of meeting these challenges—both related to each other in ways where false boundaries between problems and solutions melt into a coherent theory of UX practice.

Once UX research is understood and accepted as a form of creative design research, it must repeat itself through shared recorded practices and values, resulting in shared resources that can inspire UX professionals and researchers through outstanding examples that significantly reshape future practices. CLUF could include a repository for such shared resources.

CLUF must create an intellectual context that is compatible with the realities of UX work in creative technology settings. JUS was formed almost 10 years ago to provide a venue for UX related research to which existing HCI venues were not favorable. JUS thus began, relative to the four phases identified above, in the transition from muddling to mastery. JUS reviewers who remain attached to the standards that created the muddling phase of UCD research can bring expectations for controlled replicable studies that are impossible when studying work that must rely on professional judgment and expertise. Such standards remain hostile to communicating mastery as evidenced in specific project contexts.

Creative UX case studies must not be subject to inappropriate standards from the earlier muddling phase of UX research. We should expect new forms of creative case studies from maestro UX professionals with drives, passions, competences, and agendas that match those of the RDIs in Davies and Talbot’s (1987) study. Such forms have recently been given novel support from new research practices within IxD (Gaver, 2011; Koskinen et al., 2011) that accept and expect the realities of creative design work, but in turn demand standards of evidence, competent critical reflections, and arguments that transform creative practice into knowledge producing research. CLUF needs to extend these practices from contemporary design research to the presentation and review of creative UX work. Drawing on research through design practices, CLUF’s ground rules should include:

• the fundamental need for high quality production values for IxD within creative UX work, irrespective of any gaps created between maestros and middleweight practitioners
• the fundamental role of competent informed critical reflection (as opposed to study planning) in creative UX work
• openness to all forms of process, and not just user-centered ones such as ISO9241-210
• judgment of creative UX work by its demonstrable replicable results, and not by the scientific validity of its practices, especially where creative leaps and insights are involved as well as associated subjective subconscious drivers

Summary

By the time JUS was established, research in support of IxD within HCI had already passed through two phases of modelling and muddling. Few fully formed and/or usable UCD practices resulted from these first two phases. Instead, UX mastery became associated with professional conferences such as U(X)PA and practitioner books. However, JUS faces potentially insurmountable challenges in becoming a channel for disseminating this mastery, as it remains attached to the scientific values that locked UCD research in its muddling phase.

Recent developments in research through design methodologies offer a new set of research—authoring, reviewing, and relating critical practices (e.g., archival, commentary, discussion)—that can now document proven mastery of the best UX practices. Constructive research through design methodologies (e.g., Gaver, 2011 and Koskinen et al., 2011) support practices that allow, and perhaps even require, maestro performances in IxD work. Such methodologies provide a middle ground between traditional scientific practices and the acclaim lavished on UX celebrities. Maestros are expected in two counterbalanced senses. Firstly, maestros are expected to exist, and not seen as a problem for valid objectively assessed replicable UX practices. Secondly, maestros are expected to burst through the bubbles of celebrity to expose and defend their practice through an informed and appropriate critical repertoire that combines evidence, argument, and vision in an appropriate balance.

There is no argument here to do away with science. All design work depends on scientific discoveries, knowledge, or practices, but only to some extent. Science is thus necessary but not sufficient for successful design. Design work is too complex and contingent for everything to be evidenced and validated, not least because the nature of wicked problems, which pervade IxD, means that the best understanding of a problem follows the acceptance of its solution. This is at odds with scientific requirements for carefully specified problems that are frozen prior to exploration of solutions that can be verified as such.

JUS can thus remain loyal to the human science origins of usability, and yet escape the muddling phase of UCD research by focusing on masterly case studies. A complementary venue, that I have called CLUF, can provide an interactive community where work of creative UX maestros can be disseminated and discussed. Such a separate multimedia interactive format is better than masterly books that are several hundred pages long. CLUF needs to be separate from JUS to ensure that reviewing is apt.

JUS has always aimed to support UX practitioners. To fulfil this aim, it should create a complementary venue that accepts, values, and promotes the creative realities of UX work. UX celebrities have had stages and blogs for this for several years now. Their keynotes, courses, and workshops increasingly incorporate creative practices that would have been shunned as subjective and unscientific in 1980s HCI and 1990s UCD, and present, but still relatively unrecognized, in UX from the 2000s.

To adequately and appropriately share leading edge UX work, a new form of curation and critique is required that builds on research methodologies from practice-led creative design research (constructive design research). As to what this should look like and how we should judge it, has to be a matter for exploration, example, and debate. There can be no methods for this, no plans, and no fixed criteria. Such luxuries are all post hoc, and will follow from the rear view mirrors of experience: methods and plans are always clearest and most complete and credible looking back, at which point their qualities finally reveal themselves. The purpose of creative studies is not to demonstrate the achievement of aims, but to identify achievements worth aiming at and the routes by which maestros get there. Creative design does not attempt to get “there” from “here,“ in the sense of Simon’s (1981) transformation of a current state into a preferred one. Instead, creative design is always “here” and the purpose of case studies is to argue for where that is and why it matters.

Well presented, well analyzed, and critically discussed case studies are vital to the maturation of creative UX work. They will let the UX profession mature into an essential respected creator of value for the software and digital sector. This industry will mature once celebrity maestros on stage are complemented in breadth and depth by masterly UX and IxD experts with unequivocal public credentials equivalent to those of leading architects and designers. Becoming an adult requires the acceptance of realities and a tempering of ideals. There is no doubt about the desirability of scientific ideals and the promise of validated UX methods with guaranteed results regardless of who is using them and where they are being used. Unfortunately, maturity and fantasies do not mix well. We can have dreams or realities, but we cannot have both when the gap between them is too wide and there is no credible route from today’s disappointments to tomorrow’s utopia. We have to accept where we are as a creative professional practice. Until we have done that, we can have no hope of moving beyond current realities to better futures.

Acknowledgements

This editorial looks to the future on the basis of the history of HCI, UCD, UX, and IxD, drawing on my personal experience from over three decades of research and practice. I have been very fortunate to have worked with many exceptional designers, UCD/UX professionals and HCI researchers. I would particularly like to acknowledge the contributions to the work referred to above made by the following groups and individuals: the project team for the BNR Aurora project (especially Tunde Cockshott); the re-design team for EPRC (University of Strathclyde) for the Fly on the Wall Exhibition at Glasgow’s 1996 Festival of Design; Commercial and other customers of UK regional and national industry support projects at the University of Sunderland (Digital Media Network and its industrial board, CODEWORKS—especially the CEO Herb Kim, HCI researcher Sharon McDonald, and the project industrial boards, HEIF Digital Knowledge Exchange); Gillian Crampton-Smith and Philip Tabor for their support for my NESTA Fellowship work; my PhD students Darryn Lavery, Steven Clarke, Alan Woolrych, Mark Hindmarch, Eamon Doherty, the late Paul Gnanayutham, Susan Jones, Brendan Cassidy, Michael Leitner, Vicky Teinaki, and Malcolm Jones; the members of European COST Actions MAUSE (2004-2009, www.cost294.org/), and TwinTide (2009-2013, http://www.twintide.org/); and many other collaborators in research and practice who I cannot list in the space available (many though are listed in the Acknowledgements at http://www.interaction-design.org/encyclopedia/usability_evaluation.html).

References

My own papers cited below can be downloaded from
https://northumbria.academia.edu/GilbertCockton

Adlin, T., & Pruitt, J. (2010). The essential persona lifecycle: Your guide to building and using personas. San Francisco, CA: Morgan Kaufmann Publishers. Beyer, H., & Holtzblatt K. (1998). Contextual design: Defining customer-centered systems. San Francisco, CA: Morgan Kaufmann Publishers.

Bucciarelli, L. L. (1994). Designing engineers. Cambridge, Massachusetts: MIT Press. Cockton, G. (2006). Designing worth is worth designing. In A. I. Mørch, K. Morgan, T. Bratteteig, G. Ghosh, & D. Svanæs (Eds.), Proceedings of NordiCHI 2006 (165-174).

Cockton, G. (2007). Make evaluation poverty history. alt. CHI paper, CHI 2007, available from http://www.academia.edu/1906725/Make_Evaluation_Poverty_History Cockton, G. (2008a). Designing worth: Connecting preferred means with probable ends. Interactions, 15(4), 54-57.

Cockton, G. (2008b). Revisiting usability’s three key principles. In M. Czerwinski, A. M. Lund, & D.S. Tan (Eds.), CHI 2008 Extended Abstracts (2473-2484).

Cockton, G. (2008c). Load while aiming; hit?. In H. Bau, S. Diefenbach, M. Hassenzahl, F. Koller, M. Peissner, & K. Röse (Eds.), Usability professionals (pp. 17-22), German UPA Chapter, available at http://www.researchgate.net/publication/43184724_Usability_Professionals_2008_Berichtband_des_sechsten_Workshops_des_German_Chapters_der_Usability_Professionals_Association_e.V_07.09.-10.09.2008_Lbeck

Cockton, G. (2013). A load of cobbler’s children: Beyond the model designing processor. CHI EA ’13 Extended Abstracts on Human Factors in Computing Systems (pp. 2139-2148). New York, NY: ACM.

Cockton, G., & Clarke, S. (1999). Using contextual information effectively in design. In A. Sasse, & C. Johnson (Eds.), INTERACT 99 Proceedings (pp. 578–585).

Cockton, G., Kirk, D., Sellen, A., & Banks, R. (2009). Evolving and augmenting worth mapping for family archives. In A. F. Blackwell (Ed.), Proceedings of HCI 2009—People and Computers XXIII—Celebrating people and technology (pp. 329–338), BCS eWIC, available at http://www.bcs.org//upload/pdf/ewic_hci09_paper42.pdf

Cockton, G., Kujala, S., Nurkka, P., & Hölttä, T. (2009). Supporting worth mapping with sentence completion. In T. Gross, J. Gulliksen, P. Kotzé, L. Oestreicher, P. Palanque, R.O. Prates, & M. Winckler (Eds.), Human-Computer Interactions—INTERACT 2009, Part II, (LNCS 5727, pp. 566–581) Springer.

Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. In A. Sasse, & C. Johnson (Eds.), Human-Computer Interactions—INTERACT 99 Proceedings (pp.347–355). IOS Press.

Cockton, G., Lavery, D., & Woolrych, A. (2003). Chapter 57: Inspection-based methods. In. J. Jacko, & A. Sears (Eds.), The Human-Computer Interaction Handbook (LEA, pp. 1118–1138).

Cockton, G., & Woolrych, A. (2001). Understanding inspection methods: Lessons from an assessment of heuristic evaluation. In A. Blandford, J. Vanderdonckt, & P.D. Gray (Eds.) People and Computers XV (pp. 171–192). Springer-Verlag.

Cockton, G., Woolrych, A., & Hindmarch, M. (2004). Reconditioned merchandise: Extended structured report formats in usability inspection, In CHI 2004 Extended Abstracts on Human Factors in Computing Systems (pp. 1433-1436). New York, NY: ACM.

Cockton, G., Woolrych, A., Hornbæk, K., & Frøkjær, E. (2012). Inspection-based methods. In J. Jacko (Ed.), The Human-Computer Interaction Handbook (3rd edition, pp. 1275–1293).

Conklin, J. (2006). Dialogue mapping: Building shared understanding of wicked problems. Chichester, England: John Wiley & Sons, Ltd.

Constantine L., & Lockwood, L. (1999). Software for use: A practical guide to the essential models and methods of usage-centered design. New York, NY: ACM Press.

Cooper. A. (1999). The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity. Sams Macmillan Computer Publishing.

Cross, N.(2011). Design thinking: Understanding how designers think and work. Berg.

Darke, J. (1979). The primary generator and the design process. Design Studies 1(1), 36–44.

Davies, R., & Talbot, R. (1987). Experiencing ideas: Identity, insight and the imago. Design Studies, 8(1), 17–25.

Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem-solution. Design Studies, 22(5), 425–437.

Dumas, J., & Saparova, D. (2012). The first seven years of the JUS, Journal of Usability Studies 8(1), 1–10.

Frøkjær, E., & Hornbæk, K. (2008). Metaphors of human thinking for usability inspection and design. ACM Transactions on Computer-Human Interaction, 14(4).

Garnik, G., Sikorski, M., & Cockton, G. (2014). Creative sprints: An unplanned broad agile evaluation and redesign process. Proceedings of NordiCHI ’14. (pp. 1125–1130). ACM.

Gaver, W. (2011). Making spaces: How design workbooks work. Proceedings of CHI 2011 (pp. 1551–1560). ACM.

Gothelf, J. with Seiden, J. (Ed.; 2013). Lean UX: Applying lean principles to improve user experience. Sebastopol, CA: O’Reilly Media.

Gould, J., & Lewis, C. (1985). Designing for usability: Key principles and what designers think. CACM 28(3), 300–311.

Gray, W. D., & Salzman, M. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. HCI, 13(3), 203–261.

Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(1), 421–443.

Holtzblatt, K., Wendell, J., & Wood, S. (2005). Rapid contextual design: A how-to guide to key techniques for user-centered design. Morgan Kaufmann.

Keinonen, T. (2009). Design method: Instrument, competence or agenda. Paper presented at the Swiss Design Research Network Symposium, Lugano, Switzerland.

Koskinen, I. K., Zimmerman, J., Binder, T., Redström, J., & Wensveen, S. (2011). Design research through practice: From the lab, field, and showroom. Morgan Kaufmann.

Law, E., Hvannberg, E., & Cockton, G. (Eds.; 2008). Maturing usability: Quality in software, interaction and value. Springer.

McGinn, J., & Kotamraju, N. (2008). Data-driven persona development. In Proceedings of CHI ’08 (pp. 1521–1524). ACM.

Molich, R. (n. d.) CUE—Comparative usability evaluation. Retrieved October 2014, from http://www.dialogdesign.dk/CUE.html.

Pruitt, J., & Adlin, T. (2006). The persona lifecycle: Keeping people in mind throughout product design. Morgan Kaufmann.

Rosenbaum, S. (2008). The future of usability evaluations: Increasing impact on value. In E. L.-C. Law, E. T. Hvannberg, & G. Cockton (Eds.), Maturing usability (pp. 344–378). London: Springer.

Schön, D. (1983). The reflective practitioner: How professionals think in action. Basic Books, Inc.

Sears, A., & Hess, D. J. (1999). Cognitive walkthroughs: Understanding the effect of task-description detail on evaluator performance. International Journal of Human-Computer Interaction, 11(3), 185–200.

Simon,H. A. (1981). The sciences of the artificial. MIT Press.

Sorell, T. (1991). Scientism: Philosophy and the infatuation with science. Routledge.

Wisser, F. S., & Stappers, P. J. (2007). Mind the face. Proceedings of DPPI ’07 (pp. 119–134). ACM.

Woolrych, A., Hornbæk, K., Frøkjær, E., & Cockton, G. (2011). Ingredients and meals rather than recipes: A proposal for research that does not treat usability evaluation methods as indivisible wholes. International Journal of Human-Computer Interaction, 27(10), 940–970.

Published in: in Volume 10, Issue 1,