Remote UX Research in Unpredictable Times: Takeaways from Early Pandemic Practices

Abstract

The recent global pandemic had shocking impacts on societies and economies worldwide by bringing complexity, ambiguity, and uncertainty for extended periods. UX researchers, faced with numerous challenges, were compelled to adapt to emergent conditions through remote research methods.

Investigating how some members of the UX research community adapted to the emergency offers valuable insights into how they responded to uncertainty and showed resilience. Therefore, we studied the early practices of UX managers and researchers in Turkey during the first 6 months of the global pandemic under lockdown conditions. Our goal was to understand how UX research and design teams responded to uncertainty by customizing their methods.

We focused on seven firms as different cases to conduct semi-structured interviews with seven managers and 13 UX researchers. The results demonstrate that the adaptable nature of UX research allowed them to quickly respond to these emerging conditions.

In the early period of the pandemic, our study participants—UX managers, designers, and researchers—experienced the advantages and challenges of remote UX research necessitated by the lockdown. These included issues related to participant recruitment, work performance, readiness, and methodological concerns. Based on our findings, we make recommendations for UX researchers to conduct moderated and unmoderated remote research while maintaining inclusiveness and rigor.

Keywords

UX research, remote UX research, COVID-19, pandemic conditions, research practice, moderated, unmoderated

Introduction

Recently, UX practice was challenged by the complexity, uncertainty, and ambiguity of the global COVID-19 pandemic. Several concerns arose.

UX Research in the Pandemic Was Remote

At its onset, social distancing measures necessitated adopting remote-only ways of carrying out UX practice, even though much of UX research has traditionally involved direct contact with users for observation and consultation. Furthermore, the pandemic had significant effects on the economy and generated a risky environment for businesses. Thus, being close to users and understanding how they adjusted to the pandemic circumstances was seen as a way to mitigate the economic risks and deal with the uncertainty (Craven et al., 2020; Diebner et al., 2020).

UX researchers and designers are always accommodating new situations and new technologies in an ever more complex environment (Norman & Stappers, 2015). Therefore, despite its challenges, conducting research to understand emerging user behaviors became a necessity (Moran, 2020), and it could only be done remotely in the earlier phases of the pandemic. Although social distancing measures are no longer a necessity, many practitioners believe that “remote UX research is here to stay” as it outperformed their expectations during the pandemic (Schumacher, 2022).

Remote UX Research Was New

Of course, remote UX research was already an established way of gathering information from users before the pandemic, especially in areas such as cross-cultural research. Many guides address its advantages and disadvantages (Bolt & Tulathimutte, 2010; Rubin & Chisnell, 2008), and there are digital tools for remote user research and testing (Whitenton, 2019). But, at the onset of the pandemic, remote research was new to many UX researchers who had to learn it while managing the unfamiliar conditions of COVID-19 and its possible effects on UX research practice.

Therefore, starting in 2020, there were efforts to guide researchers in converting face-to-face practices into remote ones (Balestrucci et al., 2020; Kondratova et al., 2021; Mathis et al., 2021; McCloskey et al., 2022; Schmidt et al., 2021). For example, Gov.uk compiled a remote research tool kit (Baron & Petre, 2020). Lupton (2020) initiated a crowdsourced document on performing remote fieldwork. And Larsen et al. (2021) conducted a workshop on sharing experiences of remote user research during the pandemic.

UX Research Methods Became Logistically Important

Many studies highlighted observations regarding the gap between UX practice and theory and how methods are used (Agogino et al., 2015; Gray, 2016a; Kramer et al., 2016; Stolterman, 2008). In addition to the existing guidelines and suggestions for conducting remote research, it is crucial to today’s understanding to gain insight into how UX researchers and designers employed and integrated design research methods into their practices during the pandemic’s emergent conditions (Gray, 2016b).

Süner-Pla-Cerdà et al. (2021) reviewed 49 blog posts from different websites posted by UX practitioners. They showed that earlier discussions among UX researchers about the effect of the pandemic on UX research practice revolved around methodological challenges and opportunities of remote research, ethical concerns about conducting research during the pandemic, and regarding the psychological effects of the pandemic on participants and UX researchers. Although reviewing these discussions is beneficial to understand the overall impact of this sudden shift to remote work, these blog posts are mostly limited to optimistic views regarding how they can sustain UX practice based on business concerns.

To understand the real experiences of UX researchers under the emergent conditions caused by the pandemic, it is vital to consult them. Investigating how resilience in crises is built into communities may provide opportunities to plan better and prepare for future situations like the pandemic (Suleimany et al., 2022).

The Pandemic Influenced UX Researchers

We studied how the pandemic influenced what UX researchers did. To understand the transition, we investigated UX research practices during the first 6 months of the pandemic by interviewing researchers about their practices at several firms that conduct UX research.

In the following sections, we briefly review the literature about current approaches to remote UX research and how researchers responded to the pandemic. Then, we introduce our method and highlight our major results. Last, we discuss the transformational effects of the pandemic on UX research practice and recommend tips for UX researchers.

Related Literature

Even though remote study became a new paradigm for many practices during COVID-19, remote research is not new to the UX community.

Remote UX Research Can Be Moderated or Unmoderated

Remote UX research is a way to conduct research in which the researcher can gather data from a participant in a different location. There are two fundamental ways to conduct remote research: moderated with the session guided by a moderator, or unmoderated (automated) with web-based tools to conduct the research without a moderator’s presence (Barnum, 2021; Bolt & Tulathimutte, 2010).

Moderated research is conducted synchronously through online tools; the participant, moderator, and observers focus on the same task at the same time. Researchers can collect behavioral information by observing users’ actions and noticing their speech patterns (Rubin & Chisnell, 2008). They can also acquire attitudinal information by talking to users and probing at emerging subjects during the session. Thus, the data are mostly qualitative (Bolt & Tulathimutte, 2010).

Unmoderated (or automated) remote research is usually carried out asynchronously. The participants’ performance of a predetermined task is recorded through certain tools, and then the researchers analyze the collected data (Barnum, 2021; Bolt & Tulathimutte, 2010; Rubin & Chisnell, 2008; Tullis & Albert, 2013). A narrowly defined task is often assigned to the participants with close-ended or very brief open-ended questions. Therefore, chiefly quantitative data is gathered, and a larger sample size can be achieved compared to moderated research (Bolt & Tulathimutte, 2010). Diary studies (Bolger et al., 2003) and cultural probes (Gaver et al., 1999) can also be conducted in a remote and asynchronous way to obtain in-depth information about the experience.

Remote Research Has Many Advantages

As remote research is independent of environment and location, it has many advantages, such as requiring relatively fewer people, being lower cost, and requiring less time (Gannon, 1998; Krauss, 2003; Varsaluoma & Sahar, 2014; Venturi, 2008). Moreover, remote research can reach wider sampling groups, especially in cases like international research or where the researcher’s physical presence is difficult or not possible (Jain et al., 2011; Walsh et al., 2010; Yiu, 2013). Such advantages enabled UX researchers to work under social distancing conditions, especially in the first months of the pandemic.

Remote Research in the Pandemic Had Challenges

As expected, the efforts to continue UX research during the pandemic were mostly centered around synchronous and asynchronous ways of conducting research through online mediums and crowd-sourcing platforms (Balestrucci et al., 2020). Researchers even reported that many user studies with hard-to-reach groups were successfully conducted through remote mediums in areas such as the defense industry (McCloskey et al., 2022), universal design (Simon-Liedtke et al., 2021), and geriatric services (Kondratova et al., 2021).

Yet a major challenge was to bring studies that normally required a physical setting out of the lab. Attempts to overcome this challenge included enabling remote procedures that normally required local hardware and software, such as neurophysiological measurements for UX evaluation (Vasseur et al., 2021) and facial expression detection in user tests (Giroux et al., 2021). Further strategies involved using existing data from web or device logs, creating prototypes that could be used online, and bringing prototypes to participants’ homes.

An emergent way to solve this challenge was to run studies in extended reality (XR) environments (Schmidt et al., 2021). XR technologies, such as virtual and augmented reality, are seen as a potential way to replace the need for real-world experimentation and tests (Wölfel et al., 2021). There were several examples of successful applications of remote XR studies carried out under pandemic circumstances (Saffo et al., 2021; Siltanen et al., 2021; Spittle et al., 2021). As there was a population of individuals who already had certain equipment to immerse in XR settings, recruitment was easy and could be fruitful for such groups (Radiah et al., 2021; Saffo et al., 2021).

However, target groups unfamiliar with XR settings or who do not own the equipment could be challenging to work with. To access diverse participants, some researchers suggested establishing hardware landing schemes (Ratclife et al., 2021) or funding hardware distribution (Steed et al., 2020). Moreover, for remote XR studies, standardizing the study procedure could be problematic as participants’ equipment differed and the application of the procedure was determined by the participant. To control for these differences, Steed et al. (2020) suggested observing the participants during the study through videoconferencing platforms.

Pandemic-Induced Concerns Remained

When such remote efforts did not work well, researchers waited for the pandemic circumstances to settle and then continued research in physical settings while complying with hygiene regulations and social distancing measures (Miclau et al., 2021). However, continuing face-to-face research during the pandemic caused concerns regarding health and safety (Steed et al., 2020), whereas research in general raised concerns about research validity (Süner-Pla-Cerdà et al., 2021). Other issues included concerns for employees’ and participants’ wellbeing with “technostress” (Ayyagari et al., 2011) and “zoom fatigue” (Fosslien & Duffy, 2020).

Despite the challenges, maintaining research efforts and reassessing the existing norms and values have been crucial for UX researchers. This will help prepare for the post-pandemic era and future changes (Balestrucci et al., 2020).

Our Study Adds to the Literature

The studies above provide inspiring and valuable insights about UX research during COVID-19. But it is also valuable to examine early practices of this transition from the practitioners’ perspectives to help understand the transformational effects of the pandemic and how researchers responded to such emergent conditions. This could help others in their current practice and help prepare for future transformational challenges.

We conducted semi-structured interviews with managers and UX researchers from seven firms about their experiences in the first 6 months of the pandemic to understand how they responded to uncertainty by customizing their methods. It provides valuable insights into the challenges and opportunities presented by unexpected circumstances, particularly in the shift toward a remote approach. By addressing these hurdles, it aims to elevate the overall quality of UX research.

Methods

Examining real-world problems requires a holistic approach that entails consideration of the environments, dimensions, and actors related to a phenomenon. This helps researchers understand the professional context, the practitioners’ ways of working, and their perspectives (Robson & McCartan, 2016). For a holistic approach, we find case studies suitable for investigating UX research practices in the industry since they allow researchers to explore real-world systems and phenomena, analyze relationships within specific contexts, and examine multiple perspectives, as noted by Yin (2018). By studying the practices of firms as cases, it is possible to understand their research process, including UX researchers’ considerations and expectations. Therefore, we investigated the early practices in the pandemic through interviews with UX researchers and managers from seven in-house teams and consultancies in Turkey. In this way, we delved into the details of the cases and analyzed them in-depth for valuable insights into the UX research practices employed in the industry.

Choosing the Country

Turkey is a representative context for this study because UX is growing there. Especially in the last two decades, Turkey has witnessed a rapid increase in interest in UX in parallel with the rest of the world. The study by İnal and Rızvanoğlu (2016) shows that the number of UX consultancy firms, specialized teams, and the number of people in the field is expected to increase in Turkey. The IT sector has shown significant growth in recent years, even with the depreciation of the Turkish Lira (TÜBİSAD Informatics Industry Assocıation, 2018).

The Ministry of Development defines IT as a lifting force as the sector is one of Turkey’s most extensive export fields (T.C. Kalkınma Bakanlığı, 2023). This development positively affected the UX service sector as IT companies heavily employ UX designers and researchers (UXservices Field Report, 2018-19). Many UX design teams and consultancy firms provided UX services and continued their practices at the beginning of the pandemic.

Finding Our Cases and Participants

We took Turkey as the sampling frame, and within this sampling frame, we employed three successive phases to define our firms by considering “eligibility” (Eriksson & Kovalainen, 2010) and the “typical case selection” criteria (Gerring, 2006).

First, we compiled a list of companies conducting UX research in Turkey. The resulting pool involved 11 UX consultancy firms and 20 in-house UX teams. From this pool, we invited six consultancy firms and 13 in-house UX teams that fit our sampling criteria to participate in our study. As a result, we were able to interview UX managers and researchers from five UX consultancy firms and two in-house UX teams.

Table 1 summarizes the characteristics of cases and participants, including the roles and experience levels of participants. All participants were involved in UX research activities during the pandemic, even though some held titles other than researcher.

Table 1. Case and Participant Characteristics

Firm	Firms’ Details	Participant	Job Role	Years of Experience
Firm A	– Develops software-based services for both international and local markets. – Mainly conducts remote UX research.	P-1	UX Research Director	3-10 years
		P-2	UX Researcher	3-10 years
		P-3	UX Researcher	0-3 years
		P-4	UX Researcher	3-10 years
Firm B	– UX design consultancy for digital and physical products. – Has little experience with remote UX research.	P-5	UX Consultant/Founder	+10 years
		P-6	UX Researcher	3-10 years
		P-7	UX Designer	0-3 years
Firm C	– UX research team of a bank service. – Did not have any experience in remote research before COVID-19.	P-8	Customer Experience Team Manager	3-10 years
		P-9	Customer Experience Researcher	0-3 years
Firm D	– Develops a remote research tool for international and local markets. – Provides UX research consultancy by using their own digital tool.	P-10	UX Consultant/Founder	3-10 years
		P-11	UX Researcher	3-10 years
Firm E	– UX design consultancy for digital products. – Has little experience with remote UX research.	P-12	UX Consultant/Founder	+10 years
		P-13	UX Researcher	0-3 years
		P-14	UX Researcher	0-3 years
Firm G	– Branch of an international design consultancy company. – Has little experience with remote UX research.	P-15	Director/Partner	+10 years
		P-16	UX Designer	0-3 years
		P-17	UX Designer	3-10 years
Firm H	– UX design consultancy for digital and physical products. – Has little experience with remote UX research.	P-18	UX Strategist	+10 years
		P-19	UX Researcher	3-10 years
		P-20	UX Designer	+10 years

Interviewing the Participants

We conducted semi-structured interviews between July and October 2020 to understand research practices and our participants’ experiences in the very early phases of the pandemic. Interview questions focused on understanding their perspectives toward UX and UX research, and their process of UX research, including commonly used methods before the pandemic and alterations after that. (See the Appendix for the interview questions.)

We conducted and video-recorded all interviews using Zoom™.

Analyzing the Data

Recordings were transcribed verbatim so we could do content analysis. Content analysis is a method to systematically examine qualitative data by coding them to reveal valid and meaningful patterns (Miles et al., 2014; Saldana & Omasta, 2017). These patterns have been used to write case reports for each firm to understand replications and differences in practices.

We used the “member checking” process to validate data (Creswell & Miller, 2000). First, we asked each participant to check their interview transcriptions, which may have potentially distressing elements. If there were such elements, we deleted them. Second, based on our content analysis results, we generated case reports for each firm and distributed them to the managers to receive their confirmation and validate our inferences.

All data gathered in the study and case reports presented to firms were in Turkish. The quotes used in the results section were translated by the authors. Three researchers independently coded the raw data in spreadsheets and then compared, discussed, and consolidated their codes to reach agreement on the coding structure. Coding was done in two cycles that revealed the related themes of the study. Themes, their definitions, and related sub-themes guiding the coding process are listed in Table 2. The resulting themes reveal issues related to the advantages and challenges of remote UX research as discussed in the next section.

Table 2. Themes and Sub-Themes Guiding the Coding Process

Themes	Themes’ Definition	Sub-Themes
Advantages of remote UX research during the early pandemic
Benefits	Advantages of remote research that emerged with the pandemic	lack of physical space restrictionsease of sample recruitmenteffects on collected data qualityease of building rapportimproving work performance increasing UX researchers’ productivity
Challenges of remote UX research during the early pandemic
Deficiencies in observational research	Lack of holistic observation due to limitations with digital tools	strategies to overcome limitations of remote research in terms of observationdependency on self-reported datalimitations regarding laboratory use and strategies to overcome these limitations
Inclusiveness for non-tech savvy users	Issues around inclusiveness for sample groups that may struggle with the technological requirements of remote research	concerns about participants’ unfamiliarity with digital toolsstrategies suggested including non-tech savvy participants
Readiness for the research sessions	Practices and concerns of researchers to be prepared for remote research sessions	preparedness through methodological planningUX researchers’ preparednessparticipant preparedness
Moderator engagement in synchronous sessions	Practices of ensuring effective moderator engagement in synchronous sessions, where moderators may struggle due to the lack of in-person interaction	strategies for building rapportconcerns about limitedness in receiving visual cluesstrategies to sensitize participants for remote sessions
Managing unmoderated sessions	Practices to ensure that participants are able to complete tasks successfully and efficiently without direct guidance from a moderator	strategies for planning unmoderated research concerns about participant representativeness in unmoderated research

Results

Like many other work areas, the pandemic significantly affected UX research practice in this early phase.

At the onset of the epidemic, five of the participant firms had to cancel or postpone projects involving physical interaction, contextual observation, or a laboratory setting (Firms A, B, D, G, and H). However, before the pandemic, Firms A and D, in particular, as well as B, E, G, and H had already been partially conducting their research through digital mediums for usability testing and for international research purposes. This experience increased their readiness and enabled a smooth transition to remote work.

Furthermore, with the sudden relocation of activities from the physical to the virtual, three of the participant firms (Firms A, D, and E) became interested in investigating the opportunity areas to develop new products or enhance the existing ones that aligned with emerging user habits during the pandemic.

Figure 1 illustrates the methods frequently used by UX teams before the pandemic, as mentioned in the interviews. Asterisks in the figure indicate methods that our participants said they also used during the pandemic. Participating firms were mainly able to conduct interviews and usability tests. Methods such as netnography and diaries were used as complementary to other methods.

Figure 1. The most frequently-used methods prior to the pandemic as reported in the interviews. Numbers indicate the number of firms using the method. (Asterisks indicate methods also used during the pandemic.)

During the pandemic, firms heavily focused on retrieving attitudinal information based on users’ self-reports. Therefore, at least some of the firms did not find it difficult to switch to online mediums to elicit users’ opinions.

“Actually, there is not such a complex situation here. You find the participants and talk to them remotely […] In the end, there is this human on the other side [of the screen], we are endeavoring to gather insights by asking the right questions.” (P-10)

Even though three firms had been actively implementing remote research before the pandemic, only one firm (Firm D) was working remotely. So, six firms (Firms A, B, C, E, G, and H) also changed their work style to remote in the early days of the pandemic. Participants from these firms saw video conferencing software and remote collaboration tools as ways to continue their work. That became the new normal. Furthermore, they found various advantages to remote work that they had not previously realized. 

Early Advantages of Remote UX Research

First, UX researchers could carry out research without physical space restrictions. This freedom had obvious logistical advantages in saving time and expenses and enabling UX researchers to conduct research without geographic restrictions. All the participant firms appreciated this advantage.

Second, they could more easily recruit and manage participants. In the beginning, the increase in time spent at home during curfews enabled users to allocate time for participation. Also, not needing to commute or travel and spending less time on data collection were other motivating factors for participants (P-2, 3, 5, 6, 9, 10, 11, and 17). Such benefits could enable recruiting hard-to-reach or normally busy user groups (Firms B, C, and D). That helped improve the sample representativeness for such groups (P-10).

Third, remote participation from home was seen as advantageous because it may positively affect the collected data quality. Some UX researchers (P-2, 3, and 8) believed that participating from home may help users act more naturally and relaxed and thus present a more honest picture of reality. Some even think that this enabled UX researchers to pick up contextual clues about users’ home environments, daily problems, lifestyle, and personal preferences that were impossible in regular face-to-face interviews, user tests, or lab settings (P-3, 8, 10, and 11).

“[Nowadays-early period of the pandemic] there are seminars, talks or happy hours [regarding UX research] over Zoom again. There, everybody complains like ‘remote study is hard,’ like ‘their kid climbs on them, and they can’t respond,’ like ‘here is an intrusion in the background,’ and so on… but I tell them ‘the product would be used in that setting, what else do you want more? You get the chance to observe’ […] I mean in ethnography, as an anthropological method, what matters is observing the setting by being there. In my opinion, video conferencing is something close to being in the setting for observation, even though we see the home partially.” (P-11)

This optimistic view about observing the sections of users’ lives was another motivation for normalizing the emerging circumstances of conducting UX research. Moreover, the solidarity evoked at the onset enabled building empathy between the UX researchers and their research participants. Participants defined building rapport during the research sessions as another factor affecting data quality.

“Actually, we had a shared problem. In a way, this positively affected the connection and communication. I know what the other side [the research participant] is tackling right now.” (P-3)

Fourth, the sudden shift to remote work obligated researchers to adopt digital tools they might not have experienced before. Throughout this adoption process, they noticed collaboration opportunities and connectivity features that would improve their work performance and make it more effective. Digital tools surely supported collaboration, especially when different researchers or stakeholders worked on the same project simultaneously. This also enabled knowledge exchange and transfer between the UX researchers, especially when one was more experienced.

“We write these [data analysis] codes and co-ops in Figma [for data analysis]. We can work together. Both [the less experienced researcher and I] can make changes on the same thing. […] We open Zoom on one side [of the screen], Figma on the other. We both connect from our Figma accounts and say ‘let’s code this like that, let’s split that code, there is such a story relevant to that code, let’s add that code under this heading’ and so on… we’re working on it like that.” (P-15)

As collaboration opportunities were enhanced, it became easier to manage ‘quick and dirty.’ For example, an entire usability testing study with its testing sessions, analysis, and reporting could be done in one day on an online whiteboard tool.

“We have usability testing workshops […] In one day, we can test with users in the morning, and in the afternoon in front of the whiteboard, we can now continue this online on Miro where we can quickly produce formal reports if they want […] Through such studies, we focus on improving the existing product.” (P-12)

This opportunistic approach was also reflected in how the data is handled. Since the data was collected, processed, and interpreted online, connectivity between the tools for these tasks gained importance. Thus, through such tools, firms aimed to increase their productivity by automatically transferring the collected and coded data for interpretation.

“In this period, we were a bit obsessed with automation. […] We are trying to increase the interaction between the tools. For example, automatically transferring all the data from AirTable to Miro as post-it notes… There is this [classical] designer pose in front of a wall, grouping post-its; we are transferring data from AirTable to Miro to replicate the online version of this pose. We generated templates. I mean, there is a template for Journey Maps, there is one for Mental Models. You know, it’s because the designer should spend less time with presentation and visual representation of research findings. Of course, things can change on a project basis. Needs can be different. But we are trying to make their lives easier with such templates.” (P-12)

Fifth, certain features of the teleconferencing software were welcomed as they allowed other stakeholders to participate in the research sessions without distracting the participant users (P-9). In addition, such digital tools and remote work considerably increased UX researchers’ productivity as the researchers spent less time commuting or socializing in the office (P-1, 2, 7, 9, 15, and 16). Conducting all activities online, the UX researchers could record all their work. This practice brought order, prevented data loss, and thus contributed to the organizational memory.

Last, all study participants seemed content with the comfort remote work facilitates and demanded its continuation.

Early Challenges of Remote UX Research

Deficiencies in Observational Research

Even though remote research brought many advantages, UX researchers still felt the need to observe context. Otherwise, it was hard for researchers to make sense of the holistic experience and generate insights (P-1, 2, 3, 5, and 6). This deficiency led them to search for other sources of information, such as asking the participant users to show their context during online interviews (P-11) or, if available, examining public camera recordings (P-19).

Also, to overcome this deficiency in the interviews or testing sessions, UX researchers stated that they planned to ask users to keep complementary diaries (P-7, 13, and 17), fill in questionnaires to assess daily mood (P-1, 2, and 13), and video record their own experiences (P-13). Such complementary information also helped users and researchers sensitize themselves to the research sessions.

UX researchers indicated that research could not be carried out within the social distancing context (Firms A, B, D, and H) for studies that required a laboratory, equipment such as eye-tracking or electroencephalogram devices, or evaluation of physical prototypes. As an alternative to laboratory testing and observations, UX researchers suggested conducting longitudinal field trials. In such studies, the products would be sent into the users’ contexts to be experienced and evaluated without the researchers’ presence. However, this usually was not the preferred method due to time and monetary constraints (P-5). Also, UX researchers found it infeasible to conduct remote research on products or systems requiring confidentiality (in terms of property rights or data privacy) because privacy could not be fully ensured when online (P-5).

Inclusiveness for Non-Tech Savvy Users

As the research sessions are mediated by digital tools, participant users’ technological literacy and access to the technology were significant concerns in remote research. UX researchers worried that special user groups, such as the elderly, children, and immigrant users who may be unfamiliar with the computer interfaces, would be difficult to study. Therefore, researchers thought inclusiveness could be problematic in remote research (P-5, 6, 8, 16, and 17). They were concerned that study samples were limited to technology-literate people, affecting sample representativeness.

“We started to demand for people [participants] who are capable of using technology […] Because, if that person cannot perform the tasks, that would sabotage the whole test and waste a lot of time, etcetera… Therefore [remote research] changed certain things for us. I mean, I feel bad about this, [but] I’m discriminating [against participants].” (P-6)

Studying users with low technology literacy puts pressure on both the participant users and researchers as these users needed constant guidance throughout the session, negatively affecting the rapport between the two.

“Sometimes, with users who do not feel competent to use the technology, it is challenging to ask them even to share their screen over Zoom. I feel uncomfortable, constantly saying things like ‘No, do this! No, no, no, not there! Not like that!’ At some point, if you empathize with the other person, I may make them feel like a child. Here, users have to feel comfortable conveying their actual thoughts. These kinds of interventions […] may make the user feel like ‘I cannot do it!’ and they can just give up.” (P-17)

To study non-tech savvy participants, UX researchers followed different strategies. They self-experienced and examined the product or system by considering the characteristics of the group (P-6). They consulted the area experts such as healthcare professionals who were closely working with such groups (P-2 and 5). They used additional methods such as observation or netnography to complement the already available information (P-13). 

Readiness for Research Sessions

UX researchers heavily emphasized that remote research required more preparation and planning than face-to-face research sessions. This preparedness involved the following: (1) methodological planning, (2) UX researchers’ preparedness to ensure their readiness, and (3) participants’ preparedness through sensitizing and informing them about the research session.

Methodological planning included planning research materials (questions, tasks, scenarios, procedures, and prototypes) and mediums (digital tools used in the research). UX researchers emphasized that all research procedures, tasks, scenarios, and questions presented to users should be easy to understand and follow, especially when the research is done through online mediums without the physical presence of a moderator (P-6, 7, 13, 17, and 19). Researchers suggested that all remote researchers should consider factors such as memory load and cognitive fatigue and keep the tasks as simple as possible by dividing them into basic steps (P-1, 2, and 19).

Moreover, the researchers warned that prototypes’ fidelity matters. The prototypes needed to work smoothly without any functionality problems because the practitioner did not have a chance to guide the participant users if something did go wrong. The ability to reset the whole interface or other interactions beyond the test scenarios can allow users to explore the interface on their own (P-16). This way, users can be more engaged with the research session.

“[As users don’t feel competent with the online mediums], they feel like the prototype is kind of an alien environment for them. Of course, we put tricks like ‘escape getaways’ for the cases where they are stuck or extra interactions outside the scenarios to provide space for them to navigate more, to try out by themselves. All these for relaxing them a bit.” (P-16)

Besides the research materials, UX researchers emphasized that research mediums (software and tools used in the sessions) needed to be carefully selected and studied for project requirements and the capabilities of the participant users (P-1, 3, 7, 13, and 19).

“The first thing I would say [to the researchers who would like to conduct remote research] is ‘Choose a tool which the user can easily use!’ Because no matter how good we are at using that tool or analyzing the data, it would be pointless if we don’t get it right.” (P-19)

Firms even made up fictional or non-profit projects to examine potential tools’ capabilities and experience them.

“There we generate an extra task for ourselves and conduct research [on a social service] just to test a remote testing tool. Indeed, [if] we received good results, I suggest the tool to the ones asking.” (P-18)

While selecting the right tool, connectivity between that tool and the other tools used in the UX research process mattered as the tools needed to smoothly transfer data and enable sharing with other stakeholders.

Despite all these preparations, remote research is always prone to technical setbacks. Therefore, UX researchers recommended having backup plans and alternative mediums by envisioning all possible setback scenarios. For the same reason, they found pilot studies essential to test all alternative mediums and plans.  

UX researchers’ preparedness was also vital for effective remote research sessions in addition to methodological planning. In remote communication, building rapport was more challenging than face-to-face sessions. UX researchers stated that making research participants feel comfortable throughout the research session was their responsibility. To do that, they highlighted the importance of practitioners’ communication skills and suggested collecting information about participant users prior to the session (P-2 and 11). This information could include demographics, previous experience with the product, and participants’ current contexts.

“It is much easier to maintain a natural conversation with the users and sensitize them to the study when we are from the same culture. However, I experience difficulty, indeed, in building such rapport with users from abroad, because I don’t know anything about the person’s context on the other side. I mean, it could be a terrible day in that country, it could be raining like hell or a disaster maybe… I have no clues.” (P-4)

Likewise, the pandemic could have damaging effects on a user’s life. Researchers considered that this would affect the research data, so it was critical to be informed about such situations prior to the research sessions. If that is the case, they recommend postponing the studies.

“[At the beginning of the pandemic], people seemed [distracted] rather than focusing on the user test we’ve been conducting with them. Each person has a worry, let me say […] If users have other things on their minds, I think user tests can be postponed for a while. Especially in times like this, when people are highly worried, I think the results can be affected to some extent.” (P-9)

Besides having background information on the participant users, the researchers needed to be prepared by setting up the equipment and checking if the tools and prototypes were functioning. Furthermore, researchers warned that working online caused more fatigue; thus, they had to consider limiting the number of sessions per day for their own wellbeing.

[Before the pandemic], we could do six to seven tests per day. [Right now], I do three in a day, and I finish the day saying, ‘Man! I’m exhausted.’ Because you need to be extra alert, extra cautious […] For the one who moderates the test, it is more tiring than the studies we normally do face-to-face.” (P-6)

Participant preparedness is also crucial for remote research sessions. Researchers recommend that preparations should sensitize the participant users and inform them about the research session. Before the session, participant users should be informed about the study content, stages, what to expect from them, and the digital tools used in the sessions. To guide participants, firms prepared hard copy handouts (B, E, and G), generated guidance pages on their websites (E), and shot instruction video guides (E).

“At a basic level, we literally guide users, as in, we are preparing a manual on downloading the application, and so on. [In face-to-face sessions] if the guy had a problem downloading, you could take the phone and download and install it for him. There weren’t any problems there.” (P-12)

Moderator Engagement in Synchronous Sessions

In moderated remote research sessions, establishing rapport with the user is crucial for maintaining data quality. To achieve this, UX researchers use a range of strategies during synchronous sessions. These may include initiating warm-up conversations, using language and a speaking style that puts the participant at ease, and demonstrating empathy throughout the study (P-2, 3, 4, 6, 7, 9, 10, 15, 17, and 18). Furthermore, it is important for researchers to hone their narrative and empathy skills, whether conducting remote or face-to-face studies (P 2, 3, 4, 7, 10, 11, and 17).

“Rapport is also an essential issue in my studies, both remotely and in-person, to grasp the person’s condition on the other side and, to some extent, to establish a language of communication. This is the most significant aspect of the interview. Because it has a significant impact on the interview’s quality.” (P4)

Despite the challenges of building rapport through digital communication, remote research offered a relaxed atmosphere for participants, as opposed to laboratory observation environments that could cause stress and unease (P-12 and 17). Moreover, some user groups may feel alienated or nervous in environments like offices where interviews and tests are conducted. UX researchers suggest that remote work is more comfortable for such groups and could potentially lead to higher data quality (P-11, 17, and 19). However, our participants noted that observing facial expressions and gestures was challenging in remote research (P-5, 6, 12, 15, 18, and 19).

In addition, sensitizing participant users to activities and materials used in face-to-face sessions wasn’t as effective, or even possible in some cases, in digital environments (P-18). As a result, UX researchers need to encourage participants to share their thoughts and feelings more during remote research sessions by engaging them in meaningful conversations (P-5).

UX researchers are also advised to pay attention to the words and accents used by participants (P-11). Be alert enough to pick up clues through the limited observation opportunities available in remote interviewing tools to develop better insights (P-1, 2, and 11).

Managing Unmoderated Sessions

In unmoderated asynchronous UX research sessions, researchers’ lack of direct control over the process raised concerns about the efficiency and effectiveness of data collection. For this reason, all participants stated that UX researchers should carefully plan the question sets and scenarios to be used in their sessions and consider the limited level of control they have over the session. Through planning, researchers can ensure that the data collected is of the highest possible quality and that their research objectives are achieved.

“In the asynchronous test setup, you cannot fix things on the road. The arrow is already released from the bow [when the data collection starts]. The flow needs to be excellent there. If there is a lack of guidance or a directing mistake, that will alter the findings, or there may be some issues with the medium.” (P18)

To ensure effective data collection, UX researchers defined step-by-step sub-tasks to reduce the cognitive load on the participants. This way, participants wouldn’t have to memorize all the tasks and give brief, straightforward responses (P-1, 2, 3, 10, and 11).

“I really try to break down the tasks [the tasks to be given to the participant during the test] question by question as much as possible. They get very confused when you expect them to do more than one thing in a question. After writing both of them, they try to do the second one, not the task you gave them. Moreover, sometimes it is necessary to give small retrospective reminders in some tasks, you know, ‘Look, you are doing something like this, so you need to do it like this’. Because it can happen, you know, participants can be quite detached from what is happening and what is over.” (P-3)

Accordingly, instructions and task descriptions given to participants needed to be clear, simple, and concise. Moreover, P-12 from Firm E recommends using short videos to guide participants through asynchronous processes.

Representativeness of the participants was also of concern in unmoderated sessions. Factors such as their motivation to participate for monetary compensation could potentially impact the quality of the information obtained (P-1). To mitigate this risk, a screening survey was conducted before asynchronous studies to ensure that only relevant and genuine users were selected to participate (Firm A). This screening process helps to exclude participants who do not belong to the targeted group, thereby ensuring that the study outcomes are not affected by irrelevant data.

“There is a point that we have noticed, especially in remote [asynchronous] user tests, one test goes very well and the other does not go well at all. And there is no specific reason. We realized that some users might be very tired and take the test. Before that, they may have taken 20 other tests and then taken our test again. To notice this, [we conduct screening surveys involving questions such as] ‘How do you assess your energy level at the moment?’ ‘How do you assess your current mood?’ [And questions on] if they’ve recently been traumatized or distracted, that affects [their responses] too.” (P-1)

Asynchronous methods didn’t always yield sufficient information as the researcher could not moderate the session (P-12). To obtain additional relevant information, complementary methods such as questionnaires (Firms A and E) or techniques like a diary or self-video recording (Firm E) were applied. These methods can help ensure that the research’s purpose and design needs are fully and accurately fulfilled.

Discussions and Conclusions

Our study addresses the experiences of our 20 participants during the early pandemic period: how they had to transform their prior UX research methods and processes, and their self-reported advantages and disadvantages during the pandemic’s emergent conditions. At that time, uncertainties and restrictions pertaining to lockdown measures made remote research the only viable UX research option. Although these restrictions were lifted, the upsurge of remote UX research is believed to be permanent (Schumacher, 2022).

A recent report by Dua et al. (2022) supports this belief with data: Remote is not only increasingly demanded by employees, but it is also being made available for jobs in the Information and Communication Technology sector. The availability of remote work and its benefits, as experienced since the onset of the pandemic, signal that remote research will become a more common practice than before; remote research may even be prioritized in many cases.

Based on our interviews during the early pandemic, our study provides answers to the following questions about remote UX research and the effects of the pandemic on our industry.

Can Remote Replace Face-to-Face UX Research?

Our findings demonstrate that the advantages of remote studies, as well as the challenges, are mostly found in moderated research. Unlike unmoderated research, which had a well-established remote methodology and tools before the pandemic, remote moderated research was new to many UX researchers. 

One major challenge UX researchers experienced was the impracticality of collecting observational data in remote research. Participants did mention picking up contextual clues during remote moderated testing, which was advantageous compared to testing in contrived lab settings. Süner-Pla-Cerdà et al. (2021) also found that UX researchers mentioned naturalistic research contexts as an opportunity during remote research. 

However, these comparisons are largely made between lab and remote settings under social distancing and even lockdown conditions. We must note that during the early period of the pandemic, UX firms and teams also had to adjust to the challenges of switching to remote work with an uncertain duration. Therefore, their first adaptation was to moderate usability testing. Observations made from a screen through a camera can be insightful but fall short of providing evidence strong enough to replace systematic inquiry of contextual factors.

Longer-term adaptation to remote settings would require incorporating asynchronous data collection methods through digital platforms. These involve such methods as diary studies and cultural probes, as well as open-ended, generative techniques such as context mapping and workshops, which are well-established research methods to elicit user experience, especially for new product development. 

All in all, given its limitations regarding contextual observation, it is difficult to confirm if remote can replace face-to-face research. Without the physical restrictions of social distancing, mixed methodologies blending remote and face-to-face settings can provide the best of both worlds. For instance, a remote diary study can be complemented by synchronous contextual research techniques. This should be done whenever contextual research is feasible and likely to reveal vital user needs. Although this can be common practice to many UX researchers globally, it might be new to those working at companies with little remote research experience prior to the pandemic. 

Is Remote More Inclusive or Less than Face-to-Face Research?

User inclusivity was another critical issue that the UX researchers raised. Since the research context and tools are limited to digital mediums in remote practice, UX researchers expressed their concerns about including user groups with low technology literacy. The challenges of diverse user representation, for example elderly users, in remote research settings are also addressed in the literature (see Kondratova et al., 2021; Richardson et al., 2020).

The inclusion of diverse user populations should be further addressed by the academic and practitioner communities to prepare for potentially restricting future situations. These communities should take action to access underrepresented users who might be hard to access even today in face-to-face conditions. A few recommendations from the literature include telephone interviews or postal surveys (Hewitt et al., 2019) and developing specialized data collection setups and delivering them to target groups (Kondratova et al., 2021). It is also essential to consult the specialists if necessary to understand and prepare for special target groups.

Another aspect of the representation of user diversity is the inclusion of differently-abled users. In this sense, remote offers potentially more inclusive research practices. Recruiting users with mobility impairments is easier in remote research rather than inviting them to the lab. Additionally, users can be observed using their own devices with mediating technologies such as software for the visually disabled. In that sense, remote ways may facilitate more inclusiveness, whereas there is room for advancement for other underrepresented groups that are not technology literate. 

Can Analysis Be More Effective Through Remote Mediums?

Most of the challenges mentioned by the UX researchers were about the preparation and data collection phases of remote UX research. The participants did not voice any specific challenges about the data analysis and interpretation phases. They did, however, acknowledge many advantages, including facilitation of collaborative data analysis and interpretation, ease of sharing, and automation features enabled by digital tools. 

Although efficiency and time-saving are highly valued in data analysis within industry settings, automation and transferring research findings as design solutions has potential drawbacks. This approach can lead to negligent or superficial practices that reduce the multi-layered, complex, and profound nature of the user experience. Although research in academia and industry may have different goals and expectations (Isaksson et al., 2020), prioritizing agility over rigor may jeopardize the research’s effectiveness.

In one of the recent issues of this journal, Barnum (2019) raised some critical concerns regarding the state of UX research by questioning current educational practices (usually in the form of short courses and camps) and automated methodological approaches to UX practice. She further challenged UX practitioners by asking whether the utility of the UX researcher is diminished with the emergence of online user research tools in which rigor is disregarded. This is a serious quandary for practicing UX researchers, especially in the post-pandemic world and for the sake of the future of the UX field.

Is UX Research Resilient to Change?

UX research is a field characterized by its adaptive nature. This is also evident in our findings which demonstrate a swift adaptation of the companies and UX researchers throughout the uncertain, emergent conditions of the pandemic. Despite the apparent challenges, participants emphasized the importance of “moving on” with the available resources and possibilities available. 

Such an agile and adaptive response was an important asset in bringing resilience to the professional UX community, which allowed them to not only survive but also become stronger during difficult and uncertain times. Furthermore, researchers highly appreciate the advantages of remote research, which is more proof that remote research will continue.

What Are the Limitations of Our Study?

Our study provides fruitful insights into the effects of the pandemic on UX research practice. However, it also has several limitations that require future studies. First, while the firms included in the study are representative of the UX field in Turkey and work in international fields, it is important to note that the results may not be generalizable to the global context and other types of industries that are not represented in our sample. Therefore, the findings should be treated as representative cases rather than universal truths.

Second, due to the pandemic, we were unable to observe participating firms’ practices in context, which limits the depth and richness of the data collected. Future studies examining UX research practices of companies should incorporate ways to overcome this limitation.

Third, our case data relies solely on self-reported views, which may be biased toward promoting the firm’s positive aspects. To address this limitation, we included UX managers and employees in the study to provide data diversity. However, further observation would increase the richness of the data and improve the reliability of inferences. 

Finally, our study focused on the overall experience of UX researchers during the earlier phases of the pandemic while there were still uncertainties. As such, we did not systematically investigate the use of data collection and analysis tools or methods. Although these tools were not new, many of these practitioners were new to using them, and their effectiveness in this context was not known yet. However, UX researchers can now reflect on their practices better since they also experienced the post-pandemic era. Therefore, a retrospective evaluation of these tools and methods can be performed to provide better insights for their improvement.

Tips for Usability Practitioners

• In uncertain circumstances like the pandemic, UX researchers would always find a way to move on and adapt to emerging conditions. This requires them to change their methods or adopt new ones as necessary. While doing this, UX researchers should be careful about the strengths and weaknesses of the methods they employ and find other ways to compensate for any weaknesses. They should employ methodological triangulation whenever necessary.
• Although many benefits of remote work settings have been experienced by practitioners since the beginning of the pandemic, without mandatory social distancing measures, it is no longer the only option. UX researchers should be aware that remote cannot replace face-to-face research for cases that require observing the context. Remote must be favored when it is the most reasonable option, not simply because it is a convenient option.
• Moderated synchronous remote sessions require more preparedness than face-to-face ones, since the moderator and the participant do not share the same context (both physical and often psychological), and the moderator does not usually have a chance to assist the participant in technical setbacks. Therefore, for such sessions, UX researchers should
  • carefully plan and test the process, tools, and materials to be used in the session, accounting for the unique characteristics of the participants involved,
  • provide comprehensive guidance and information (through video or text-based guides to outline the session content and expectations) before the session due to limited or in-direct communication with participants, and
  • familiarize themselves with the characteristics of the sampling group to adjust their communication style according to the needs of that group. 
• Unmoderated asynchronous remote sessions can be more challenging in terms of retrieving reliable information and having control over the sampling data to ensure participant representativeness. In such cases, UX researchers should collect complementary information in various ways, such as delivering screening questionnaires on current mood and circumstances, raising follow-up questions, defining post-task steps, and requesting video or audio recordings on these tasks. By doing so, researchers can maintain their control over sampling and enhance their understanding of the participants’ experiences.
• UX researchers should pay attention to the technological literacy level of their sample when studying underrepresented user groups. They should be aware that such groups are usually the most vulnerable ones during such uncertain conditions. Thus, UX researchers should consider designing inclusive data collection tools and protocols by considering varying levels of experience with technology. They should also integrate alternative data collection methods if digital tools fail at maintaining inclusiveness.
• Automation in data analysis and ease of transferring research findings as design solutions are tempting for UX researchers. However, validity and reliability must still be of concern to glean meaningful design decisions addressing representative user needs and expectations. Therefore, UX researchers should be cautious using automation in their analysis. They should not sacrifice rigor for the sake of agility because it may jeopardize the research’s effectiveness.

Acknowledgements

We would like to thank the participating firms for the invaluable knowledge they shared. We also would like to thank our fellow colleague, Hilal Şahin, for her assistance in the data collection and processing phases. This work is funded by TÜBİTAK (The Scientific and Technological Research Council of Turkey). Grant number: 120K215. 

The Human Research Ethics Committee of METU approved the data collection procedure and research instruments, including the interview questions, under protocol number 149-ODTU-2020.

References

Agogino, A. M., Beckman, S. L., Castaños, C., Kramer, J., Roschuni, C., & Yang, M. (2015). Design Practitioners’ Perspectives on Methods for Ideation and Prototyping. International Journal of Engineering Education, 32(3), 1428–1437.

Ayyagari, R., Grover, V., & Purvis, R. (2011). Technostress: Technological Antecedents and Implications. MIS Quarterly, 35(4), 831858. https://www.jstor.org/stable/41409963%0AJSTOR

Balestrucci, P., Angerbauer, K., Morariu, C., Welsch, R., Chuang, L. L., Weiskopf, D., Ernst, M. O., & Sedlmair, M. (2020). Pipelines Bent, Pipelines Broken: Interdisciplinary Self-Reflection on the Impact of COVID-19 on Current and Future Research [Position Paper]. Proceedings – 8th Evaluation and Beyond: Methodological Approaches for Visualization, BELIV 2020, 11–18. https://doi.org/10.1109/BELIV51497.2020.00009

Barnum, C. M. (2019). The State of UX Research. Journal of Usability, 15(1), 1–7.

Barnum, C. M. (2021). Usability Testing Essentials: ready, set…test! (2nd ed.). Morgan Kaufmann.

Baron, N., & Petre, L. (2020). User Research and COVID-19 : Crowdsourcing tools and tips for remote research. User Research in Government. https://userresearch.blog.gov.uk/2020/04/02/user-research-and-covid-19-crowdsourcing-tools-and-tips-for-remote-research/

Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616. https://doi.org/10.1146/annurev.psych.54.101601.145030

Bolt, N., & Tulathimutte, T. (2010). Remote research: Real users, real time, real research. Rosenfeld Media.

Craven, M., Liu, L., Mysore, M., & Wilson, M. (2020). COVID-19 : Implications for business [Executive Briefing]. McKinsey and Company. https://www.mckinsey.com/business-functions/risk/our-insights/covid-19-implications-for-business

Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory Into Practice, 39(3), 124–130. https://doi.org/10.1207/s15430421tip3903_2

Diebner, R., Silliman, E., Ungerman, K., & Vancauwenberghe, M. (2020). Adapting customer experience in the time of coronavirus. McKinsey & Company. https://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/adapting-customer-experience-in-the-time-of-coronavirus

Dua, A., Ellingrud, K., Kirschner, P., Kwok, A., Luby, R., Palter, R., & Pemberton, S. (2022). Americans are embracing flexible work—And they want more of it. McKinsey & Company. https://www.mckinsey.com/industries/real-estate/our-insights/americans-are-embracing-flexible-work-and-they-want-more-of-it

Eriksson, P., & Kovalainen, A. (2010). In A. Mills, G. Durepos, & E. Wiebe (Eds.), Encyclopedia of Case Study Research (pp. 93–96). SAGE Publications, Inc. https://doi.org/10.4135/9781412957397

Fosslien, L., & Duffy, W. (2020). How to Combat Zoom Fatigue. Harvard Business Review Home. https://hbr.org/2020/04/how-to-combat-zoom-fatigue

Gannon, P. (1998). Remote possibilities. Communications International (London), 25(5), 41. https://doi.org/10.7748/ns.29.21.26.s30

Gaver, B., Dunne, T., & Pacenti, E. (1999). Cultural probes. ACM Interactions, February, 21–29.

Gerring, J. (2006). Case study research principles and practices case. Cambridge University Press.

Giroux, F., Léger, P. M., Brieugne, D., Courtemanche, F., Bouvier, F., Chen, S. L., Tazi, S., Rucco, E., Fredette, M., Coursaris, C., & Sénécal, S. (2021). Guidelines for collecting automatic facial expression detection data synchronized with a dynamic stimulus in remote moderated user tests. In Kurosu, M. (Ed.) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 12762). Springer International Publishing. https://doi.org/10.1007/978-3-030-78462-1_18

Gray, C. M. (2016a). It’s more of a mindset than a method. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 4044–4055. https://doi.org/10.1145/2858036.2858410

Gray, C. M. (2016b). What is the nature and intended use of design methods? Proceedings of DRS2016: Design + Research + Society – Future-Focused Thinking, June, 1–14. https://doi.org/10.21606/drs.2016.307

Hewitt, J., Pennington, A., Smith, A., Gething, S., Price, M., White, J., Dewar, R., & Carter, B. (2019). A multi-centre, UK-based, non-inferiority randomised controlled trial of 4 follow-up assessment methods in stroke survivors. BMC Medicine, 17(1), 1–8. https://doi.org/10.1186/s12916-019-1350-5

İnal, Y., & Rızvanoğlu, K. (2016). Türkiye’deki kullanici deneyimi profesyonellerinin profili, çalişma ortamlari, kullandiklari yöntemler ve mevcut iş akişi içindeki konumlari. Galatasaray Üniversitesi İleti-ş-Im Dergisi, 25, 181–181. https://doi.org/10.16878/gsuilet.283063

Isaksson, O., Eckert, C., Panarotto, M., & Malmqvist, J. (2020). You need to focus to validate. Proceedings of the Design Society: DESIGN Conference, 1, 31–40. https://doi.org/10.1017/dsd.2020.116

Jain, J., Innes, J., Churchill, E., Lund, A., Courage, C., & Rosenberg, D. (2011). Managing global UX teams. CHI 2011, 527–530.

Kondratova, I., Fournier, H., & Katsuragawa, K. (2021). Review of remote usability methods for aging in place technologies. In Gao, Q., Zhou, J. (Eds.) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 12786). Springer International Publishing. https://doi.org/10.1007/978-3-030-78108-8_3

Kramer, J., Agogino, A. M., & Roschuni, C. (2016, August 21). Characterizing competencies for human-centered design. Proceedings of the ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, North Carolina, USA, Vol. 7. https://doi.org/10.1115/DETC2016-60085

Krauss, F. S. H. (2003). Methodology for remote usability activities: A case study. IBM Systems Journal, 42(4), 582–593. https://doi.org/10.1147/sj.424.0582

Larsen, L. B., Øvad, T., Nielsen, L., & Larusdottir, M. (2021). Remote user testing: Experiences and trends. In Ardito, C. Lecture Notes in Computer Science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Vol. 12936), 579–583. https://doi.org/10.1007/978-3-030-85607-6_81

Lupton, D. (2020). Doing fieldwork in a pandemic – Google docs [Crowd-sourced document]. https://docs.google.com/document/d/1clGjGABB2h2qbduTgfqribHmog9B6P0NvMgVuiHZCl8/edit#

Mathis, F., Zhang, X., O’hagan, J., Medeiros, D., Saeghe, P., MCgill, M., Brewster, S., & Khamis, M. (2021). Remote XR studies: The golden future of HCI research? Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 3.

McCloskey, M. J., Turner, K. L., Kancler, D. E., & Zelik, D. J. (2022). Supporting remote experimentation through computer-mediated communication. Ergonomics in Design, 2020, 1–8. https://doi.org/10.1177/10648046221075942

Miclau, C., Demaeght, A., & Müller, A. (2021). Empirical research as a challenge in day-to-day teaching during the pandemic of 2020/21 – Practical solutions. In Nah, F.FH., Siau, K. (Eds.) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 12783). Springer International Publishing. https://doi.org/10.1007/978-3-030-77750-0_40

Miles, M. B., Huberman, M. A., & Saldana, J. (2014). Qualitatie data analysis (3^rd ed.). SAGE Publications, Inc.

Moran, K. (2020). COVID-19 has changed your users summary. Nielsen Norman Group. https://www.nngroup.com/articles/covid-changed-users/

Norman, D. A., & Stappers, P. J. (2015). DesignX: Complex sociotechnical systems. She Ji: The Journal of Design, Economics, and Innovation, 1(2), 83–106. https://doi.org/10.1016/j.sheji.2016.01.002

Radiah, R., Mäkelä, V., Prange, S., Rodriguez, S. D., Piening, R., Zhou, Y., Köhle, K., Pfeuffer, K., Abdelrahman, Y., Hoppe, M., Schmidt, A., & Alt, F. (2021). Remote VR studies: A framework for running virtual reality studies remotely via participant-owned HMDs. ACM Transactions on Computer-Human Interaction, 28(6). https://doi.org/10.1145/3472617

Ratclife, J., Soave, F., Bryan-Kinns, N., Tokarchuk, L., & Farkhatdinov, I. (2021). Extended reality (xr) remote research: A survey of drawbacks and opportunities. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1-13. https://doi.org/10.1145/3411764.3445170

Richardson, S. J., Carroll, C. B., Close, J., Gordon, A. L., O’Brien, J., Quinn, T. J., Rochester, L., Sayer, A. A., Shenkin, S. D., van der Velde, N., Woo, J., & Witham, M. D. (2020). Research with older people in a world with COVID-19: Identification of current and future priorities, challenges and opportunities. Age and Ageing, 49(6), 901–906. https://doi.org/10.1093/ageing/afaa149

Robson, C., & McCartan, K. (2016). Real world research: A resource for users of social research methods in applied settings (4^th ed.). John Wiley & Sons Ltd.

Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2^nd ed.). Wiley Publishing, Inc.

Saffo, D., Bartolomeo, S. Di, Yildirim, C., & Dunne, C. (2021). Remote and collaborative virtual reality experiments via social VR platforms. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1-15. https://doi.org/10.1145/3411764.3445426

Saldana, J., & Omasta, M. (2017). Analyzing documents, artifacts, and visual materials. Qualitative Resarch: Analyzing Life, 63–88.

Schmidt, A., Alt, F., & Mäkelä, V. (2021). Evaluation in Human-Computer Interaction – Beyond lab studies. Proceedings of the 2021 CHI: ACM CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 1(1), 1–4. https://dl.acm.org/doi/10.1145/3411763.3445022

Schumacher, R. M. (2022). Four ways the pandemic has permanently changed UX research. UXmatters. https://www.uxmatters.com/mt/archives/2022/03/four-ways-the-pandemic-has-permanently-changed-ux-research.php

Siltanen, S., Heinonen, H., Burova, A., Palma, P. B., Truong, P., Opas, V., & Turunen, M. (2021). There is always a way: Organizing VR user tests with remote and hybrid setups during a pandemic—Learnings from five case studies. Multimodal Technologies and Interaction, 5(10). https://doi.org/10.3390/mti5100062

Simon-Liedtke, J. T., Bong, W. K., Schulz, T., & Fuglerud, K. S. (2021). Remote evaluation in universal design using video conferencing systems during the COVID-19 pandemic. International Conference on Human-Computer Interaction, 116–135. https://doi.org/10.1007/978-3-030-78092-0_8

Spittle, B., Xu, W., Frutos-Pascual, M., Creed, C., & Williams, I. (2021). Socially distanced: Have user evaluation methods for immersive technologies changed during the COVID-19 pandemic? 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 415–420. https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00094

Steed, A., Ortega, F. R., Williams, A. S., Kruijff, E., Stuerzlinger, W., Batmaz, A. U., Won, A. S., Rosenberg, E. S., Simeone, A. L., & Hayes, A. (2020). Evaluating immersive experiences during Covid-19 and beyond. Interactions, 27(4), 62–67. https://doi.org/10.1145/3406098

Stolterman, E. (2008). The nature of design practice and implications for interaction design research. International Journal of Design, 2(1), 55–65.

Suleimany, M., Mokhtarzadeh, S., & Sharifi, A. (2022). Community resilience to pandemics: An assessment framework developed based on the review of COVID-19 literature. International Journal of Disaster Risk Reduction, 80(August), 103248. https://doi.org/10.1016/j.ijdrr.2022.103248

Süner-Pla-Cerdà, S., Töre Yargın, G., Şahin, H., & Danış, S. (2021). Examining the impact of Covid-19 pandemic on UX research practice through UX blogs. International Conference on Human-Computer Interaction, 579–592. https://doi.org/10.1007/978-3-030-78221-4_40

T.C. Kalkınma Bakanlığı (Ministry of Development). (2018). On Birinci Kalkınma Planı (2019-2023) Bilgi ve İletişim Teknolojileri Özel İhtisas Komisyonu Raporu [Turkey Information and Communication Technologies Specialization Commission Report of the Eleventh Development Plan, 2019-2023]. Retrieved September 2022 from https://www.sbb.gov.tr/wp-content/uploads/2020/04/Bilgi_ve_IletisimTeknolojileriOzelIhtisasKomisyonuRaporu.pdf

TÜBİSAD Informatics Industry Assocıation. (2018, August). Why invest in Turkish ICT sector. https://www.tubisad.org.tr/tr/images/pdf/ict_sector_report_new_ispat.pdf

Tullis, T., & Albert, B. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Morgan Kaufmann.

UXServices Field Report. (2018). Turkey user experience (UX) 2018/2019. UXServices. Retrieved November 2021 from https://uxservices.com/wp-content/uploads/2019/02/TUX-2018-19.pdf

Varsaluoma, J., & Sahar, F. (2014). Measuring retrospective user experience of non-powered hand tools: An exploratory remote study with UX curve. MINDTREK 2014 – Proceedings of the 18th International Academic MindTrek Conference, 40–47. https://doi.org/10.1145/2676467.2676485

Vasseur, A., Léger, P.-M., Courtemanche, F., Labonte-Lemoyne, E., Georges, V., Valiquette, A., Brieugne, D., Rucco, E., Coursaris, C., Fredette, M., & Sénécal, S. (2021). Distributed remote psychophysiological data collection for UX evaluation: A pilot project. International Conference on Human-Computer Interaction, July, 255–267. https://doi.org/10.1007/978-3-030-78462-1_19

Venturi, G. (2008). Remote, synchronous user research for international usability: A case study. UPA Europe 2008, 1–6.

Walsh, T., Nurkka, P., & Kujala, S. (2010). Exploring cross-cultural UX evaluation with the remote online sentence completion method cross-cultural design. Proceedings of the Ninth International Workshop on Internationalization of Products and Systems, 11–22.

Whitenton, K. (2019). Tools for unmoderated usability testing. Nielsen Norman Group. https://www.nngroup.com/articles/unmoderated-user-testing-tools/

Wölfel, M., Hepperle, D., Purps, C. F., Deuchler, J., & Hettmann, W. (2021). Entering a new dimension in virtual reality research: An overview of existing toolkits, their features and challenges. 2021 International Conference on Cyberworlds (CW), 180–187. https://doi.org/10.1109/CW52790.2021.00038

Yin, R. K. (2018). Case study research and applications: Design and methods. SAGE Publications, Inc.

Yiu, C. (2013). UX design with international teams: Challenges and best practices. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2333–2336. https://doi.org/10.1145/2468356.2468759

Appendix

Interview Questions:

Q1. First, please tell us about the [remote and one-to-one] UX research studies you are currently conducting.

1. Considering the current conditions, are there any studies you have canceled because you could not run remotely in this process? Are there any studies you had to carry out face-to-face, even if it includes risk? What were the reasons for applying this way? What were your expectations from the customer/other departments in this study?
2. Have there been any studies you decided to conduct remotely instead of canceling in this process? How did you choose to conduct these studies remotely? Can you explain? What were your team’s expectations from the customer/other departments in this case? (If it was planned as face-to-face and had to be changed to remote research) What kind of decisions did you make while adapting these studies to remote ways? What kind of changes have you made in your plans?

Q2. Could you briefly explain a typical planning phase for the UX research process? How do you make decisions at this stage? Please briefly describe your preparations before the research process.

1. (If typically they do not conduct remote research) How do you plan remote UX research? Can you describe your preparations before starting the remote UX research process? 
2. (can be probed) Due to the Covid-19 pandemic, did you have to make any special preparations for your research?

Q3. Following the planning stage, can you provide a brief overview of your typical data collection process?

1. When conducting UX research, how do you define your user sample and what methods do you use to recruit them? Which channels, mediums, or methods have you found to be effective in reaching your desired participants?
2. When using remote research methods for UX research, are there any differences in how you define the user sample compared to face-to-face settings? Furthermore, which mediums or tools do you find most effective when contacting participants in remote research studies?
3. What data collection tools did you use in your face-to-face research?
4. What data collection tools did you use in your remote research? Would you be willing to demonstrate some of these tools? Additionally, are there any data collection tools that you typically use but are unable to use while working remotely (or vice versa)?
5. What were the reasons for choosing these particular data collection tools, and how do you think they could be improved?
6. What changes and adaptations did you make to your research methods to make them appropriate for remote data collection? Specifically, how did you modify the process for remote research compared to face-to-face research?
7. Have you developed any specific strategies or practices for conducting remote UX research? If so, can you explain them and discuss their effectiveness?
8. [For those who wish to conduct research during the COVID-19 pandemic], do you have any advice or suggestions based on your experience adapting research processes for remote data collection?

Q4. Considering your typical UX research, could you briefly explain the methods and tools in the analysis phase? Could you explain the analysis stage of your UX research?

1. Are there any differences between remote and face-to-face UX research regarding the analysis phase?

Q5. How do you present and communicate the user information after the analysis process?

1. How do you present your results to your clients/other departments? 
2. Can you explain how clients or different departments implement presented user information into their activities? In your opinion, how does your firm or the client firm use this information? Additionally, what steps do you think could be taken to increase the effectiveness of this implementation?
3. To what extent does remotely collected data impact the presentation of research results? Have you implemented any specific strategies to address this challenge?
4. If you had to present your research results remotely during and after the COVID-19 pandemic, how did this affect your data presentation process? Furthermore, do you think that sharing the data online impacted the overall presentation process?

Q6. Can you describe a previous case where remote UX research methods were not used but could have been? In this case, what strategies would you employ to transition to remote UX research methods, and which specific methods would you consider using?

1. Can you recall a previous UX research case that was not feasible to conduct remotely? What factors led to the conclusion that remote research methods were unsuitable for this study? What modifications or adaptations would need to be made to make the research feasible for remote data collection?

Q7. Given the opportunity to work face-to-face, have you learned anything from your experience with remote research methods during the COVID-19 pandemic that you can transfer or apply to your face-to-face research approach?Q8. Lastly, would you like to add any further insights related to remote UX research methods or tools or offer tips for UX practices in the industry?

Published in: in Volume 19, Issue 1,