Introduction
Our field faces threats to its legitimacy that we will have trouble dealing with if we continue to allow a split between academia and practice. Many academics face obstacles in the way of more “relevant” research that would be more meaningful to practitioners, and practitioners tend to see academic research as not relevant to them in reinforcing their professional identity, skills, and legitimacy as the organizational experts in user research. Meanwhile, many practitioners feel they are threatened by a perceived commoditization and co-opting of user-centered design (UCD) and user experience (UX) work. These two trends or tendencies are rarely mentioned together. How they are related, and how we address these issues will have a significant bearing on whether our field will continue to advance or whether it will wither.
In this essay I first explore the different dynamics of the worlds of academia and product development and how these differences affect the nature of the work we do. Then, I show why the world of practice needs an infusion of academic rigor that can only come from changes in the nature of academic research and the adoption of a more academic style of critical thinking in the world of practice.
In fairness, I fully acknowledge that the following analysis makes generalizations for which there are many exceptions. There are certainly academics who have made major contributions to practice and are committed to relevance, and there are practitioners who are thoughtful and rigorous in their thinking. There are also people in academia whose roles are structured so that they have some similarities to practitioner roles, and conversely people in industry who do function more like academics (such as people in industrial R & D groups). But one way or another, I maintain that even most exceptional people have to contend somehow with the problematic dynamics I describe in this essay.
Academia versus Practice
As an applied field, we are not alone in experiencing a tension between academia and the world of applied practice. For example, in medicine, doctors need the fundamental knowledge that comes from research such as that on, say, evolving antibiotic resistance in microbes due to the over-prescription of antibiotics. However, that research may provide little guidance for them on how to deal with anxious patients who are demanding antibiotics. Doctors may feel pressure to produce a perceived result by writing a prescription rather than explaining why it is not a good idea, be concerned about potential complaints by a dissatisfied client, or have a fear of liability if they do not provide an active treatment. How they actually act will depend on a complex combination of rewards and incentives, past history, social dynamics of the clinic, and experience.
In our own field of usability and UCD, the tension or gap between academia and practice has a long history. The challenge of the relationship between academics and practitioners is a perennial issue at our sister organization, Special Interest Group on Computer Human Interaction (SIGCHI), a part of the Association for Computing Machinery (ACM), where there have been major efforts to ensure that the ACM Conference on Human Factors in Computing systems (known as CHI) meets the needs of academics and practitioners, but it is still difficult to bring these two communities together. Indeed, the Usability Professionals’ Association (UPA) emerged partly from a community of professionals who felt the need for a practitioner conference distinct from CHI.
As Avi Parush (2006) points out, many practitioners engaged in design, evaluation, and/or implementation of technology feel that academic research is not useful to their day-to-day life in companies. He quotes an unnamed practitioner as saying: “There are very few, if any, research articles published in scientific and academic journals that can be utilized effectively in the practice of HCI design” (p. 61).
In this journal, Caroline Jarrett (2007) described how to write research papers that appeal to practitioners. In general, practitioners look for and need research when they encounter a knotty problem that they need to solve or when they are approaching a new situation and are looking for guidance. As Jarrett pointed out, for practitioners, “research reading generally has to have a business purpose” (p.1). She gave a number of excellent suggestions to researchers to help them explain their research in more practitioner-friendly ways, and this is certainly a good start, but for many practitioners, research in general is still rarely relevant to them. This suggests that the problem is not only with how research is written, but also in what they say and whether the pre-occupations of researchers are helpful to practitioners.
Incentives and Pressures in the Academic Context
Why is it so hard for so many academics to do research that practitioners will consider relevant? The answers come from understanding the incentives and dynamics that influence the career paths of academics.
In academia, the most basic measures of success are typically scholarly publishing and obtaining grants. It is a truism that faculty must “publish or perish.” Anything that increases success in obtaining grants and producing publications in refereed publications increases career success. The quickest and easiest types of papers to write tend also to be the most narrow—and unfortunately, these are also the least likely to be considered relevant and useful by practitioners. Operationalizing sticky, real-world problems is difficult and messy. Conducting rigorous research in the real world is extremely challenging logistically. It is also much easier to get funding and to be productive if you buy into an existing research tradition and extend it through a modest variation on what has been done before. These are the social pressures towards “normal science” that Thomas Kuhn discussed in his classic, The Structure of Scientific Revolutions (1996).
When academic research takes the form of experimentation, academics are likely to focus on isolating variables to discover the abstract relationships among them, using artificial simulations. Research that involves proof of concept for new forms of human-computer interaction usually is also carried out in an artificial context and with a very narrow focus. A noted researcher acknowledges the pressures toward a narrow and artificial focus in his blog, writing the following:
“The reviewers [for the CHI conference which is considered by academics to be among the most prestigious venues for publication] simply do not value the difficulty of building real systems and how hard controlled studies are to run on real systems for real tasks. This is in contrast with how easy it is to build new interaction techniques and then to run tight, controlled studies on these new techniques with small, artificial tasks” (Landay, 2009).
The goal of tenure adds to the pressures that skew the choice of research topics. The emphasis on volume of publications is especially true for young faculty who, in order to achieve tenure, must churn out papers and get grants. Therefore, the pressure to do research that is tightly focused and relatively easier to publish can push young faculty to do less relevant research. This becomes a habit over the 7 years leading to tenure decision, and as we all know, longstanding habits are very hard to break. This same dynamic influences graduate students who may be interested in academic careers, because there is also increasing competition for academic posts in our field. Also, graduate student research must be circumscribed so it is feasible for the student to graduate in a reasonable time.
Most academics will spend much of their careers investigating a particular problem area, with increasing depth and focus—and some would say narrowness—over time. In addition, one’s colleagues at other universities reinforce this specialization by partnering on publications, presentations at conferences, and grant proposals. It is almost unheard of for an academic to switch to an entirely different field (e.g., to move into Humanities from Computer Science) and such a switch would require significant re-tooling, additional education, and an excellent rationale to be taken seriously.
In none of the above do I mean that academics are lazy or avoiding complex problems. Rather, they are simply following the natural incentives and shared interests of their community. Work driven by theoretical interest in the nature and influence of particular variables naturally pushes for narrowness of focus—isolation of variables through controlling confounds. Operationalizations of independent and dependent variables can only capture particular manifestations, and in the choice of operationalization, the emphasis is on ease of measurement.
However, increasingly, students are finding that jobs in academia are scarce. Many academic programs now have terminal master’s degrees or industry-focused programs that are intended to prepare students for life as practitioners. While some of these include teaching by practitioners, most do not, although many include internships or other industry experience as part of the program.
While academic research may not be seen as relevant, it is usually rigorous. Academics deepen their understanding by applying rigorous analysis and thought to decoding underlying principles and developing theories to explain their findings. This focus on rigor is a hallmark of the best academics. They know their research will be exposed to criticism by specialists from their own discipline. While this may promote caution in the topics taken on, it has the virtue of incentivizing methodological soundness and critique.
Incentives and Pressures in the World of Practice
The messy world of practice is replete with interesting, relevant, and knotty problems and challenges. In contrast to isolating variables to study them for their own sake, every product development decision reflects the intersection of and tradeoffs among huge numbers of variables. The challenges of product development in the business world are thus inherently extremely difficult to address rigorously. Furthermore, research in the business world takes place in a culture where there are unfortunately many disincentives to the rigorous and critical thinking at which our academic colleagues excel.
Rather than “publish or perish” as the guiding principle, practitioners live by the mantra “produce or perish.” The goal of their work is the integration of user-centered information into the development of products or services so these will be useful and usable. Therefore, the information practitioners seek about users, users’ context, and the usability of products are all intended to support those products or services, rather than to satisfy an intellectual interest. Personal career success depends on the perception that you make a valuable contribution to helping products progress through the pipeline, and that your contribution increases the chance of commercial success—preferably that you are indispensable to this. Deepening knowledge of the relationships among abstract variables is of little use in itself and does not increase your standing. Focusing on these without tying the dialogue back to concrete product-development decisions can interfere with career success by making you seem too “academic,” which is an epithet in many corporate settings.
The context of product development has a tendency to suppress skepticism and critical thinking. This is not to say that product ideas can’t be questioned, but there are always issues that the team feels it has already made and moved on from. Questioning these can lead to be seen as obstructionistic. When the consensus begins to move in a certain direction, there can be intense pressure on practitioners to be silent about their qualms. Practitioners often must demonstrate enthusiasm at least about the basic product concept. Cognitive dissonance that inhibits questioning grows as more time and energy are invested in the product.
There are many examples of products where everyone other than the people on that team wonder how on earth the product has gotten so far down the pipeline, because of “obvious” flaws. For example, we once did ethnographic research on consumer behavior for a team working on a battery-powered machine to take care of odors in the refrigerator. It was extremely obvious early on that this concept made no sense to consumers because it was solving a problem that consumers did not perceive that they had. Either they did not perceive odors in their fridge (even when we did), or they had other, cheaper solutions (e.g., a box of baking soda) that worked fine for them. They saw no reason to buy a costly machine requiring batteries to do the job that the baking soda did just fine. The team, however, was passionate that this product was terrific and maintained that we had simply chosen the wrong people to visit. It was only a year and many millions of dollars later that the product was finally cancelled, due, it would seem, to “lack of a market”—something we had told them a year earlier.
Practitioners live in highly pressurized settings where decisions have to be made quickly, often without as much information, time for planning methodology, or analysis as would be ideal. The time to think deeply about findings and interpretation is greatly compressed. In fact, taking time in this way is a negative because the focus is on speed, not deep understanding. As a colleague of ours at a large company quipped, “Close enough is close enough.” Time is measured in hours or days, sometimes in weeks, rarely in months or quarters, and almost never in years. In these circumstances, practitioners must find ways to gather, analyze, and present data quickly, efficiently, cogently, and compellingly in ways that “work” in their organization.
One manifestation of the time pressure is the need to present “topline” findings before doing a full analysis of data, often within one or two days of completing the data collection. This is a symptom of an environment where the audience is also under tremendous time pressure and cannot take the time to wrestle with nuances. Someone has to take responsibility to be prescriptive and make strong recommendations, and putting too much emphasis on describing nuances can seem like hedging.
The implications of the fact that the audience for practitioners is not composed of specialists in UX and UCD cannot be over-emphasized. UCD professionals in companies work in a highly cross-disciplinary environment. This has many benefits, such as fostering cross-disciplinary learning, but it also presents many challenges. Colleagues in other, related disciplines are also trying to have influence with decision makers and experience the same pressures toward prescriptive simplification. Success can be determined not by who has the “best” methodology or analysis, but by who best understands the concerns of the decision makers and communicates in the most concise, compelling, memorable, and simple prescriptive ways. People talk of the “PowerPoint culture” that focuses on form and brevity rather than content and rationale. Career success is heavily influenced by the judgment of people outside of the profession. UCD practitioners cannot assume that these people know of or are interested in the fine points of methodology that allow well-trained UCD people to make a contribution that is distinct from that of other people who bring in data about users and customers or recommend aspects of product strategy based on beliefs or information about people (marketing, sales people and field representatives, customer support people, etc.).
Another implication of working in a cross-functional environment is that it is important to maintain collegial relationships and choose one’s battles. This means that there is a tendency to defer to others rather than confronting underlying paradigm differences or critique each other’s methods.
Another factor that comes into play is the relative fluidity of the UX practitioner’s career. Rarely does someone stay in the same job or even the same specialty for an entire career the way an academic typically does. People often move around—both within a company, between companies, and to new areas of specialty. For instance, in the 9 years of my own career prior to becoming a consultant, I worked in two different companies and did Human Factors research for the military; analyzed user manuals; conducted ethnographic research in a variety of settings; worked in the corporate IT department; developed ergonomic guidelines; created a cross-functional methodology to incorporate organizational factors in the design and implementation of internal IT systems; developed user interfaces of thermostats, control systems, and other products (consumer products, industrial controls, forms, online applications, etc.); did user testing; ran focus groups; managed a systems training group; created and ran a usability lab; did organizational development consulting; and finally, directed an integrated UX team. In addition, some of these were individual contributor roles while others were management roles at a variety of levels from first-line supervisor to middle manager to executive. There was a thread there, of course, but each new job was radically different from the last and almost every transition was with a new group and required that I learn new skills. This is hardly unusual in our field. As a result, practitioners often are in roles for which they lack formal training and must rely instead on learning on-the-job.
Why Does this Matter?
I am not articulating the differences between academia and industry because I believe mutual understanding is a good thing (although it is). Rather, it is because I think that bridging the gulf is crucial to the future of our field, because it will strengthen the quality of both academic and applied work, and help us deal with trends in the market that can dilute our contribution.
For example, just as UX practitioners are moving around in their careers, so too are people moving into UX and UCD roles from many other areas. Sometimes they have had at least some formal preparation in the form of a course or seminar, but often, they have not. Instead, just like the moving UX pro, they learn by doing. This means that, while there is the benefit of new perspectives being infused into UX, there is also the risk that the field will lose its distinct identity.
Related to this is the commoditization and co-opting of UX by other disciplines. In the past, usability could lay claim to a professional identity based on a particular set of practices or methodology—usability testing. Of course we have more to offer than just critiquing other people’s designs, including applying our research skills to deeper levels of product definition. However, as usability has moved out of the usability lab and the usability profession has expanded its mandate to look not just at usability (narrowly defined as the ease of accomplishing a defined goal) but also at defining what those user goals are, understanding the context of use, looking at perceived value and emotion, its overlap with other disciplines has increased.
For instance, marketing people in companies and in agencies are increasingly selling usability and other user experience research services, or sometimes co-opting them under other names, but with a not-surprising tendency for them to resemble the methods that come out of the intellectual culture of marketing research, with its heavy reliance on self report. Marketing groups typically have larger budgets and hold management’s ear in a way that HCI people often don’t. This creates the risk of decreasing the perceived value and distinctiveness of the usability profession. Competition for the turf of introducing understanding of users into product development means that professions and vendors are under pressure to differentiate themselves. This leads to proliferation of branded research methods being sold on the basis of their intuitive appeal to decision makers who may well not have the knowledge or inclination to carefully weigh their validity. This environment promotes a proliferation of methodological fads and of pundits fighting for visibility by making provocative, memorable, and simplistic pronouncements that undermine disciplined and nuanced thinking. These trends raise the question of what is our profession’s intellectual core. What defines usability and UX as a body of knowledge and a profession at all? If we do not have a strong and constantly updated conceptual base, why is our opinion any more valid than anyone else’s? How do we justify our particular approaches to observation, collection of data, analysis, or understanding of users as a basis grounded in research?
Part of the answer is an infusion into industry of the type of critical thinking associated with academia, and of practice that is backed up by sound research. By critical thinking I mean being willing to ask things like the following and being prepared to evaluate the answers:
- Course of action X implies assumptions Y and Z. How do we know them to be true, and what are the risks if we are wrong?
- Is this a data-based decision or a philosophy-based decision? Is there data that could help?
- What is the nature of the data? How was it collected? Was the methodology sound?
- Whether the data is qualitative or quantitative, was it analyzed rigorously or merely impressionistically and taken at face value?
- Does the data really support this interpretation or allow us to draw this conclusion? Have we ruled out other reasonable and important alternative interpretations? What other research would do so?
For academic work in our field to play this role of supporting practice, it needs to focus on things that matter to practitioners, but it is not only the practitioners who will benefit. Such “engaged scholars” will find a wealth of extremely interesting, perplexing, and complex problems to research—great grist for those who want to make a difference in the world. Researchers who get too locked into “normal science” elaboration of existing research paradigms will miss out on new research questions of interest that will be raised by the evolution of new technologies. The following are some other areas that come to mind:
- Studies of methodological validity under realistic conditions
- Empirical assessments of common product development practices and approaches
- A paradigm for case studies that lets us draw generalizable lessons from rich experiences in the trenches
- Once we have this, we need to test some of these generalizations
- Longitudinal studies of the process and dynamics of product adoption and the behavior change that goes with it, because many business decisions are based on intuitive psychological theories of how masses of people change behavior to adopt technology
Academics benefit in practical ways from closer connections with the commercial world as well. In some companies, there are opportunities for academics to partner with practitioners to do research that would not be possible by either side alone but that can answer real needs for information. Practitioners can sometimes help provide access to key populations of interest and can help leverage corporate resources to provide funding for certain types of research activities. In addition, deeper understanding within academia of the world of practice can help academics do a better job of preparing their students to function as professionals in the face of the pressures they will encounter as they increasingly go into jobs in industry rather than in academia. Alliances can help provide students with real-life experience that makes them more marketable in tough economic times and can also provide industry with thoughtful practitioners who understand both the rigor of academia and the pace of industry. In short, both academics and practitioners stand to benefit from strengthening the bridge between academia and practice.
The Role of UPA
The challenge of the relationship between academics and practitioners is a perennial issue at our sister organization, SIGCHI. It is less of an issue within UPA. This is partly an accident of history. UPA broke away from SIGCHI specifically because there was a sense that practitioners were not getting the kind of information they needed at CHI. As a result, our organization was largely focused on practitioners from the very beginning. As more academics become involved in UPA, it is always possible that similar tensions will arise. However, I believe that UPA has an opportunity to demonstrate how academics and practitioners can work together collaboratively to create the kinds of win-win situations that provide us all with the benefits that our differences can bring. The more we really are engaged scholars and thoughtful practitioners, the larger the positive impact our field can have. I think that is worth aiming for.
References
Jarrett, C. (2007, November). Problems and joys of reading research papers for practitioner purposes. Journal of Usability Studies, 3 (1), 1 – 6.
Kuhn, T. (1996). The structure of scientific revolutions. Chicago, IL: University of Chicago Press.
Landay, J. (2009, November 7). I give up on CHI/UIST. Blog entry, Retrieved November 10, 2009, from http://dubfuture.blogspot.com/2009/11/i-give-up-on-chiuist.html
Parush, A. (2006, November/December). Practice and research in human computer interaction: towards a common ground. Interactions, 13(6), 61-62.
