The Field of UX Research Is on the Rise
These are booming times for the field of UX research. Looking broadly at the field of UX professionals, Jakob Nielsen (2017) estimated growth from about 10,000 people in the late 1990s to about 1,000,000 in 2017. Data from Google Trends indicates that UX research has grown even more rapidly than the broader UX field in recent years: Searches for the term “UX research” have suddenly risen several-fold since early 2016, while over the same period searches for UX have less than doubled. The result of this growth is clear—UX research is now a large field, with about 36,000 professionals on LinkedIn[1], and a popular one, ranked as the 39th best job in America by CNN and Payscale (Braverman, 2017). Job postings on the site Indeed portend that that the field will continue to grow rapidly, with over 1,000 open positions currently listed[2].
Investment in UX research by large technology firms is driving a substantial portion of this growth. For example, Google has over 1,100 researchers and Facebook over 750—meaning more than 5% of the field works at these two companies alone—and Microsoft has over 500[3]. The size of the teams at these and other large technology companies such as IBM, Amazon, Spotify, and Twitter have led to growing sophistication. For example, in addition to the standard embedded researcher structure, where a researcher or research team is dedicated exclusively to a single product area, it is now not uncommon for individuals or teams at large companies to focus on an important business goal, user type, end-to-end flow, or research method that cuts across multiple product areas. In another sign of growing organizational sophistication, rather than simply having researchers report to designers or product managers as is common in smaller organizations, companies with large teams now have multiple levels of researcher hierarchy. Researchers report to research managers, who report to research directors, who may even report to a research VP, allowing for extended research career paths and greater penetration of research values into company leadership. Perhaps most notable has been the exciting rise at many companies of ResearchOps—a whole field dedicated not to conducting UX research directly, but to supporting UX research by managing recruiting and logistics, vetting software and vendors, and building out infrastructure, like research repositories (Metzler, 2020). Clearly, as these teams approach previously unimaginable sizes, they are able to push the boundaries of how UX research exists in an organization in a multitude of innovative ways.
At the same time, the reach of the field continues to expand outside of large technology companies. While most UX researchers continue to work for the biggest technology firms, full time researchers are increasingly found in other sectors of the economy as well. UX research is now common in areas as diverse as financial services, healthcare, insurance, government, and retail. My personal favorite example of how much the field is broadening came from a researcher I met a few years back who was employed by a construction company. He explained how his job differed from mine by saying, ‘’you study how people use apps and websites; I study how people dig holes.” When construction companies are investing in UX research, it is clear that we are no longer a niche field.
The remarkable growth of the field also can be seen in the surging interest in formal and informal training opportunities (Okan, 2018). As a faculty member in the Master of Applied Psychology Program at the University of Southern California, I have witnessed this growth firsthand in the changing nature of our applicants. When I joined the faculty three years ago, students rarely mentioned UX research as a career interest in their application materials. Today many do, and up to 20% of my students have already held a role—often an internship, but sometimes a full-time position—in UX research before they take my UX research class. In addition to academic programs, industry bootcamps and certificate programs have proliferated, offered by academic institutions, for-profit companies, and individual practitioners (Barnum, 2019). There are now numerous ways to obtain training to enter this field, and interested entrants are availing themselves of them.
As a final indicator of the growth of the field, consider the growth in specialized UX research software. Every year, the company UserTesting publishes an account of the UX research tools ecosystem. In 2019 they noted 90 platforms. In the 2020 edition, they included 137 (Allen, 2021). While many of these tools, such Qualtrics and Zoom, are not specific to UX research, quite a few are. You can now choose between numerous platforms dedicated to UX recruiting, moderated sessions, unmoderated sessions, diary data collection, and insights management. And these are not all small offerings created by some UXR’s engineering buddy on the weekends. According to Crunchbase, platforms such as User Interviews, Lookback, and dscout have all attracted millions of dollars in funding each, while UserZoom and UserTesting have attracted tens of millions of dollars each. Recently, legendary venture capital firm Andreessen Horowitz indicated enthusiasm for investing in the user research technology market (Li, 2021). Clearly, entrepreneurs and venture capitalists have noticed the remarkable growth of UX research as well.
The Field of Usability Research Is in Jeopardy
UX is a broad area, encompassing a person’s perceptions and responses when using or anticipating use of a product, system, or service (Bevan, 2009). An important component of UX is usability (Rusu et al., 2015). Usability is generally understood to describe how well users can achieve specified goals when interacting with a product, system, or service, excluding the dimensions of affect, sensation, meaning, and value that are included in UX more broadly (Law et al., 2009). While the prospects for the field of UX research as a whole appear as bright as ever, I fear that the future of usability research specifically may be dimmer.
Looking at the history of the term “usability” in Google Trends, we see a consistent decline in interest over the duration of the dataset, which stretches back to 2004. This stands in marked contrast to the increasing interest seen in UX research over that time. I would argue that this is not merely a case of changing semantics. Indeed, it is consistent with my observation that many companies are pulling back from conducting rigorous usability testing, even as they are investing more in UX research overall.
I previously mentioned the increase in the number of in-house UX researchers at many companies. What I did not previously mention is that at the same time as these companies are hiring many researchers as employees, they are also bringing on large numbers of contractors through staffing agencies and outsourcing firms. These contract and temp staff are typically recent graduates or people transferring from another career with little prior professional experience in usability research (a point which was noted in passing in Siegel and Dray, 2019). They are usually placed into limited-time positions of one to two years, after which the company’s internal guidelines require that they leave. During the time they are on staff, they meet the team’s usability needs while the permanent researchers tackle foundational research. If the most well-staffed teams are relying on the most junior and transient members of their teams to conduct usability research, it sends a clear message to stakeholders and leadership that usability research is easy to do, less important, or both. Indeed, numerous research team managers have told me that they have been directed by their leadership to have their full time, senior staff focus on foundational research rather than usability research.
An extreme version of this philosophy is evident in the “democratize research” movement (Barnum, 2019). This movement posits that UX research in general, and usability research in particular, need not be executed by an expert with rigorous training and extensive practice. Instead, the argument goes, with guidelines and support, anyone can conduct effective usability research. Often this means the UX designer conducts research on their own prototypes—an approach Bill Albert has humorously called “the fox guarding the usability lab” (Albert, 2015). As at least one academic study attests, the results are predictably biased. This is true even when the designers have some formal training and experience with usability testing (Friess, 2011).
If one accepts that usability research does not require expertise, a natural next step for cost conscious leadership is to ask to what degree the research process can be automated. The most successful software offerings in our space have focused on unmoderated testing—that is, replacing the moderator of a traditional usability study with automated software (for a comprehensive list, see Allen, 2021). Further savings can be achieved by combining junior researchers or non-researcher product team members with such automated software, and indeed many of the junior roles I see my students take are focused on conducting analysis of unmoderated data. In fact, I am familiar with multiple product teams at Fortune 500 companies that have come to rely entirely on unmoderated testing analyzed by junior researchers or designers, with no traditional, moderated testing conducted at all.
Taken together, these trends suggest that the profession of usability research may be in jeopardy, as decision makers increasingly pursue lower cost options and usability research is seen as not a career path in itself, but simply an entry level step in a UX research career path.
What Should We Do?
I hope I have convinced readers of this journal that there is reason to be concerned. While the broader field of UX research is undoubtedly waxing, the specific field of usability research may be waning. If the trends I’ve highlighted above augur a period of decline in our field, what, if anything, should we do?
There are a number of compelling answers to this question. Communication is clearly important. Usability researchers need to make clear to stakeholders that good usability research is hard to do, highlighting the careful adherence to best practices that sets the work of a skilled professional apart from that of a junior researcher or automated solution. Additionally, we need to consider the career pressures faced by individual practitioners. UX researchers in positions of leadership need to ensure that excellent usability research will advance a practitioner’s career, as too often UX researchers believe that only foundational research will have the impact needed to advance one’s career at a large tech company (Giff & Dogan, 2016).
I support these approaches. But they rely on an important assumption: that sophisticated usability research conducted by a seasoned professional does indeed provide value commensurate with its cost. If that is the case, clearly what we need to do is make sure stakeholders realize that. But what if this assumption is wrong? What if stakeholders have correctly realized that a lower cost junior researcher or designer conducting research with automated tools can identify usability challenges just as well as a senior researcher using traditional methods—or at least well enough, such that any additional benefit of the more sophisticated approach is not worth its cost?
If it is true that investing in sophisticated usability research does not deliver value commensurate with its cost—or if this belief has become so widely held that it may as well be true—then I propose the best way for us to respond to this challenge is to push ourselves to evolve as a field. Let us challenge ourselves to build on our successes and develop a more mature approach to our craft. Let us create and promote a way of conducting usability research that identifies more of usability problems our research is currently overlooking. Let us justify the higher cost of using experienced professionals and non-automated methods with a higher return on the investment we are asking our stakeholders to make in us.
I see four important steps we can take to evolve our field so that we identify more usability problems and justify the investment a rigorous, professional approach requires:
- We can promote use of more sophisticated existing methods and which academic research has shown will lead to the discovery of more usability errors.
- We can improve usability research education.
- We can promote a stronger academic knowledge base of how to conduct usability research most effectively.
- We can use the growing body of academic knowledge established in Step 3 to inform our practices, pushing us to become an evidence-based field.
I will spend the rest of this essay briefly outlining each of these steps.
Using More Sophisticated Methods
A first step to improving usability research will be to use more sophisticated approaches that research has shown will identify more usability problems. Critical to this will be recognizing the limits of the very small sample sizes (5 to 6 participants) that are common in the field. In fact, the field has been grappling with the limitations of small samples for as long as it has been advocating for small samples. In his industry-shaking 1989 paper, appropriately titled “Usability Engineering at a Discount,” Jakob Nielsen argued that usability tests could effectively detect errors with as few as three people (1989). He noted that additional errors would be found with more participants, but the benefit of running more participants declined sharply after those first three, and thus it might make sense for cost-conscious product teams to stop at that point. This stance made perfect sense at the time it was published, as usability testing was comparatively rare and expensive, design best practices and usability were comparatively worse, digital experiences were used by far fewer people, and companies lacked the tremendous resources that the large technology firms possess today. In this context, Nielsen was not arguing against testing more, but rather he was arguing against not testing at all—the default for most product teams in the 1980s.
Fast forward to 2021, we rarely find ourselves arguing against the idea that no testing should be done at all. Furthermore, platform recruiting technologies and remote video conferencing have made testing with a large sample much faster and cheaper. Given these shifts, I believe it is time to revisit the orthodoxy of small samples. Of course, several researchers have emphasized the value of larger samples in the decades since the ascendency of Nielsen’s Discount Usability movement took hold, with Spool and Schoeder (2001) arguing that research with 5 participants only identified about 35% of usability problems in certain conditions, and Molich (2018) claiming that even research with hundreds of participants will only reveal a small fraction of usability problems on complex websites. Rather than continuing to follow the standards of decades ago, let us push to increase our sample sizes and start detecting the subtler usability problems that Nielsen and others wrote off as too expensive and difficult.
Another key step will be to better “stress test” the products and services on which we conduct usability research by more thoughtfully choosing whom our samples contain. Self-assessed digital skills vary widely across the population of internet users, even in developed countries, and these assessments predict digital behavior (Levin & Redmiles, 2021). Today, many usability researchers exclude from participation those with low self-assessments of digital skill. What if we took the opposite approach and weighted our samples towards less-skilled individuals? Given that such individuals are perhaps the most likely to experience usability challenges, this approach could make quite a bit of sense. A similar benefit could be achieved by ensuring that our samples include representation of people with disabilities, a population that is estimated to be greater than 20% of the adult population in the United States (Brucker & Houtenville, 2015). For example, individuals with low vision might struggle to find a low contrast button, or individuals with a motor control condition might struggle to click in a small response area. People without disabilities may experience some challenge in these conditions as well, but the researcher may fail to identify them as a usability problem if the challenge does rise to the level of a delay or error. It is also worth recognizing the moral imperative of accessibility research, given that people with disabilities often rely even more heavily on technology than non-disabled individuals to complete everyday tasks.
Here I have noted just two approaches, both of which are relatively straightforward to implement. Organizations with the funds to do so should explore others that are supported in the literature, such as using multiple moderators and multiple session analysts, both of which have been shown to increase identification of usability problems (Hertzum et al., 2014; Sauro, 2018).
Improving Usability Education
Another important step to improve the outcomes of usability research will be for those of us involved in usability research education to train our students better. This is true for independent training programs, bootcamps, and certificate programs, and most of all for Master’s degree programs, which are becoming increasingly common entry points into the field. We need to ensure that when our students leave us, they will propagate best standards, take pride in continuing to evolve the field, and most importantly provide more value to their organizations. Because the discussion amongst educators about what standards academic programs teaching user research should adhere to is still in its infancy (Rodwell, 2021), let me suggest that we begin with a focus on two key areas.
First, we need to improve how we teach usability moderation. Moderation lies at the heart of most forms of rigorous usability testing. Yet, I see that professional UX researchers often fail to rigorously adhere to best practices in moderation, an observation that has been made in prior research on the topic (Boren & Ramey, 2000; Molich et al., 2020). If we want to ensure our students are able to find as many usability challenges as possible, we should start by improving their ability to perform this core activity. As a model, we can look at how clinical psychology programs train their students in psychotherapy. Doctoral training and certification require thousands of hours of supervised clinical practice after classroom training in therapy technique is complete. While it may not be practical in our field for students to receive supervisory review on a large number of moderated sessions during their education, I wonder how many professional researchers have ever had even a single session of moderation rigorously reviewed? I suspect the number is small.
Second, while our field is an applied one and our students come to us almost exclusively to seek professional training, we need to make sure we steep them in the relevant academic literature. Critically, this should include the small body of work that focuses specifically on usability, but it should also include literature from the broader set of applied fields from which we draw most heavily, including applied psychology, anthropology, and human-computer interaction. The initial purpose of this effort would be to reinforce the best practices we teach, but the bigger opportunity is to foster the questioning and analytical mindset that is the unifying goal of all higher education. If we want to continuously maximize the value we deliver to our stakeholders, we need to continuously question our methodological assumptions and those of our colleagues. Then, when questions are raised, we need to turn to the literature to find the best possible answers. This does not appear to be a common approach on most corporate UX research teams today.
Promoting Academic Usability Research
As a third key step to improve the outcomes of usability research, we need to promote a stronger academic basis for our methodologies. There are many assumptions made in our field that would benefit from more rigorous investigation. While in the short term we need to focus on ensuring adherence to established best practices in our field, in the long term our field needs to rigorously assess these best practices through academic research. At present, academic research in our field is sorely underfunded. UX research practitioners can help by lobbying our companies to support more academic usability research. Practitioners and academics alike can encourage our professional associations, like UXPA, the sponsor of this journal, to begin the process of lobbying government funding agencies, which are the source of most academic research funding in the US (Lanahan et al., 2016), to support usability research. It may also be possible to cultivate individual and private foundation support in areas of usability that foster equity and inclusion, such as accessibility research.
Becoming an Evidence-Based Field
As the academic body of work exploring usability research increases, we need to leverage it to evolve into an evidence-based field. This will mean discarding those best practices that research tells us don’t really matter and embracing new ones that do. Furthermore, we need to challenge one another when claims are made about best practices in the workplace, in thought leadership in the field, and most of all, in journals like this one. As a model, we can look to the emergence of the evidence-based medicine movement. While scientific research had served as a strong influence on clinical medical practice since the 1700s, a new generation of physicians in the 1970s and 1980s noticed that a great deal of clinical practice was still based on personal opinion and intuition (Eddy, 2011)—a situation perhaps not unlike the one we find the practice of usability research in today. In response, these physicians argued that medical practice should more closely align with the always-evolving scientific literature, changing their practices as medical researchers discovered opportunities for improvements. Beginning in 1990, the approach quickly became dominant in the field (Zimerman, 2013). Recognizing that many clinicians lacked the skill to assess the scientific literature, and that most lacked the time to do so at the scale of modern research, in 1993 the Cochrane Collaboration was started to create professional, comprehensive, unbiased, medical reviews of the literature on specific topics. Perhaps now is the time for our field to embrace evidence-based usability research, with UXPA taking the lead on organizing panels of experts that publicly assess the literature on areas of importance to our community?
Conclusion
I sincerely hope this challenge will push us to evolve into a more sophisticated and effective field, one that finds more usability problems in the experiences we study and thus does an even better job of promoting a more usable world. And if we discover that great usability research is actually as easy as some think and that even subtle usability problems can be found by junior researchers, minimally trained designers checking their own work, or unmoderated platforms, then I welcome that discovery. If companies can increase the usability of their products and services more cheaply and easily, then the world will surely become more usable and better for all of us. However, as usability professionals and scholars, we have an obligation to rigorously interrogate the hypothesis that conducting great usability research is simple. Even more importantly, we have an obligation to continue to evolve our field and to use its insights to make our world more usable.
Footnotes
[1] Search was conducted in April, 2021, using the following Boolean term in the Title field of LinkedIn search, with results restricted to People: “(UX OR User OR Design OR Usability) AND (research OR researcher)”. While this search clearly results in some erroneous results, it also misses individuals who do not use LinkedIn, as well as individuals who hold a title like “Research Manager” which does not specifically indicate a UX focus, so it is probably a reasonable estimate.
[2] The following search was conducted on Indeed in April, 2021, in the What search field: “title:((UX or User or Design or Usability) (research or researcher))”.
[3] These searches were conducted in the same timeframe and using the same methodology as described in the other footnotes, with results restricted to each company. For Facebook, results included Facebook, Whatsapp, Instagram, and Oculus VR; for Google results included Google, Waymo, Verily, and Fitbit; for Microsoft, results included Microsoft, Github, and LinkedIn.
Acknowledgements
I’d like to thank my wife, Jennifer Silvers, who provided feedback on several earlier versions of this essay as well as the childcare that allowed me to finish in advance of the deadline, and Bill Albert, who provided helpful feedback but no childcare.
References
Albert, W. (2015). The fox guarding the usability lab. Journal of Usability Studies, 10(3), 96–99. https://uxpajournal.org/the-fox-guarding-the-usability-lab/
Allen, J. P. (2021, February 11). The 2020 essential UX research tools map. User Interviews. https://www.userinterviews.com/blog/ux-research-tools-map-2020
Barnum, C. (2019). The state of UX research. Journal of Usability Studies, 15(1), 1–7.
Bevan, N. (2009). What is the difference between the purpose of usability and user experience evaluation methods. Proceedings of the Workshop UXEM, 9(1), 1–4.
Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261–278. https://doi.org/10.1109/47.867942
Braverman, B. (2017, January 5). 2017 best jobs in America. CNNMoney. https://money.cnn.com/gallery/pf/2017/01/05/best-jobs-2017/
Brucker, D. L., & Houtenville, A. J. (2015). People with disabilities in the United States. Archives of Physical Medicine and Rehabilitation, 96(5), 771–774. https://doi.org/10.1016/j.apmr.2015.02.024
Eddy, D. M. (2011). The origins of evidence-based medicine: A personal perspective. AMA Journal of Ethics, 13(1), 55–60. https://doi.org/10.1001/virtualmentor.2011.13.1.mhst1-1101
Friess, E. (2011). Discourse variations between usability tests and usability reports. Journal of Usability Studies, 6(3), 102–116.
Giff, S., & Dogan, H. Z. (2016, July 1). UX research is dead. Long live UX research! Proceedings of the 30th International BCS Human Computer Interaction Conference. https://doi.org/10.14236/ewic/HCI2016.61
Hertzum, M., Molich, R., & Jacobsen, N. E. (2014). What you get is what you see: Revisiting the evaluator effect in usability tests. Behaviour & Information Technology, 33(2), 144–162. https://doi.org/10.1080/0144929X.2013.783114
Lanahan, L., Graddy-Reed, A., & Feldman, M. P. (2016). The domino effects of federal research funding. PLOS ONE, 11(6), e0157325. https://doi.org/10.1371/journal.pone.0157325
Law, E. L.-C., Roto, V., Hassenzahl, M., Vermeeren, A. P. O. S., & Kort, J. (2009). Understanding, scoping and defining user experience: A survey approach. Proceedings of the 27th International Conference on Human Factors in Computing Systems – CHI 09, (pp. 719–728). https://doi.org/10.1145/1518701.1518813
Levin, N., & Redmiles, E. M. (2021). Understanding the global landscape of digital skill on Facebook [Preprint]. SocArXiv. https://doi.org/10.31235/osf.io/a2bw4
Li, J. (2021, April 6). The market for user research platforms. Andreessen Horowitz. https://a16z.com/2021/04/06/the-market-for-user-research-platforms/
Metzler, B. (2020). Leveling up your research and research operations: Strategies for scale. Ethnographic Praxis in Industry Conference Proceedings, 2020(1), 203–217. Leveling Up Your Research and Research Operations: Strategies for Scale – EPIC (epicpeople.org)
Molich, R. (2018). Are usability evaluations reproducible? Interactions, 25(6), 82–85. https://doi.org/10.1145/3278154
Molich, R, Wilson, C., Barnum, C., Cooley, D., Krug, S., Martin, B. A., Patrowicz, J., & Traynor, B. (2020). How professionals moderate usability tests. Journal of Usability Studies, 15(4), 184–209. https://uxpajournal.org/wp-content/uploads/sites/7/pdf/JUS_Molich_Aug2020.pdf
Nielsen, J. (1989). Usability engineering at a discount. Proceedings of the Third International Conference on Human-Computer Interaction on Designing and Using Human-Computer Interfaces and Knowledge Based Systems (2nd Ed.), 394–401.
Nielsen, J. (2017, December 24). A 100-year view of user experience. Nielsen Norman Group. https://www.nngroup.com/articles/100-years-ux/
Okan, L. (2018). UX education: The rise of educational programs. User Experience Magazine, 18(5). https://uxpamagazine.org/ux-education-2/
Rodwell, E. A. (2021). A pedagogy of its own: Building a UX research program. Practicing Anthropology, 43(2), 17–21. https://doi.org/10.17730/0888-4552.43.2.17
Rusu, C., Rusu, V., Roncagliolo, S., & González, C. (2015). Usability and user experience: What should we care about? International Journal of Information Technologies and Systems Approach, 8(2), 1–12. https://doi.org/10.4018/IJITSA.2015070101
Siegel, D., & Dray, S. (2019). The map is not the territory: Empathy in design. Interactions, 26, 82–85. https://interactions.acm.org/archive/view/march-april-2019/the-map-is-not-the-territory
Spool, J., & Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. CHI ’01 Extended Abstracts on Human Factors in Computing Systems – CHI ’01, 285 (pp. 285–286). https://doi.org/10.1145/634067.634236
Sauro, J. (2018, March 27). MeasuringU: How Large Is the Evaluator Effect in Usability Testing? Measuring U. https://measuringu.com/evaluator-effect/
Zimerman, A. L. (2013). Evidence-based medicine: A short history of a modern medical movement. AMA Journal of Ethics, 15(1), 71–76. https://doi.org/10.1001/virtualmentor.2013.15.1.mhst1-1301
