Cross-National Usability Study of a Housing Accessibility App: Findings From the European InnovAge Project

Abstract

The present usability study concerns an evaluation of an Android based mobile application (app) targeting housing accessibility issues, in a prototype stage for a touch tablet. The objective was to identify and compare elicited usability issues involving senior citizens in two different national contexts: Sweden and Latvia. Applying a problem discovery approach, the study included individual sessions in home environments and follow-up user forums; researchers and user panel participants collaborated to formatively evaluate the app. To support decision-making, the task for the participants was to process information about their present or projected future functional capacity and environmental barriers in their current dwelling. The task was also to make use of a database of professionally assessed available apartments that could be considered for relocation. Data generated in the two countries were analyzed and compared.

The results generated new knowledge on relevant usability issues in the two countries and were used to provide design recommendations for the app. The identified usability issues point to a need for a more thorough exploration of design solutions that are tailored for the targeted group of people aging with functional limitations. The results also highlight how national and cultural contexts may influence perceived usability and acceptance and thereby indicate the need to consider these contexts early on and throughout the development process.

Tips for usability practitioners: Be aware of possible cross-national differences and handle those challenges in a structured and predefined way, and use real data and real sites to reach close and first-hand experiences.

Keywords

aging, culture, functional capacity, ICT products, housing provision, mobile applications, user interface, user involvement, social innovation, software development

 

Background

The following sections discuss the aging population and the accessibility of housing for certain populations, the usability and relevance for different national contexts, attempts to involve senior citizens in research and innovation projects, and our research objective for this study.

Population Aging and Housing Accessibility

Populations are rapidly aging and more people are expected to live longer periods of their lives with disabilities and reduced functional capacity (Newman & Cauley, 2012). In many respects, these trends imply major societal challenges. One area that has come increasingly into focus is the need to address housing conditions for senior citizens to maintain an independent daily life even when their functional capacity is reduced. Although the current legislation in many European countries places high demands on accessibility for the design and construction of new dwellings, a considerable proportion of the older population are living in dwellings built before such legislation was in place (Granbom, Iwarsson, Kylberg, Pettersson, & Slaug, 2016; SOU, 2015). This implies that many of those aging with functional limitations live in dwellings with stairs at the entrance, without ramps or elevators, and with many environmental barriers both indoors and in the immediate exterior surroundings. There is thus a pressing need to avoid unnecessary troubles and costs at the individual and societal levels through improved information and raised awareness about housing accessibility for informed decision-making, planning, and preventive actions among senior citizens.

Housing accessibility is important for all people and a basic prerequisite for participation in society. It is important for families with children, senior citizens, people with functional limitations, informal caregivers, and for people with temporarily reduced functional capacity. Environmental barriers threaten mobility, activity, participation, and health. The importance of housing that enables independent living increases with age as people spend more time and do more activities at home (Haak, Fänge, Iwarsson, & Dahlin-Ivanoff, 2007). Dwellings that allow senior citizens to live independent lives longer benefit the individual, their family members, and society. Accessible housing may prevent falls and injuries and reduce the need of care and isolation of senior citizens as well as delay relocation to assisted living. This can lead to benefits by improving human well-being and by reducing costs for service delivery and housing adaptions. Difficulties for informal caregivers to combine employment with caring responsibilities may also be avoided.

Usability and Relevance in Different National Contexts

To address the challenges described above, innovations that provide information and raise awareness about housing accessibility are badly needed. The growth of information and communications technology (ICT) has opened new ways for innovations in terms of communication, outreach, and scaling. However, for an ICT tool to be applicable across Europe, it is important to compare usability and relevance of the tool in countries with different building traditions, legislation, policies for housing provision for senior citizens, housing markets, and housing standards. The present paper deals with two rather different countries in this respect: Sweden representing a Nordic welfare state and Latvia representing a new EU member state.

The availability and quality of publicly funded care services related to housing issues vary across Europe. Citizens’ opportunities to apply for housing adaption grants, which are based on individual needs to enable them to remain living in their current dwelling as well as the amounts granted, differ across the EU. There are also differences within countries. As an example, 7.6 housing adaption grants per thousand inhabitants in Sweden were approved in 2014 compared to 0.1 per thousand inhabitants in Latvia (The National Board of Housing, Building and Planning, 2015). Sweden has national regulations for housing adaptations grants that are binding for all municipalities, while in Latvia, only three out of the 118 municipalities in the country provide housing adaptation grants.

Eurostat statistics on living conditions showed that in Latvia about 65% of the population lived in apartments in multi-family housing (Living conditions in Europe, 2014). In Sweden, more than half of the population (51%) lived in single-family houses. In 2014, Latvia had one of the highest overcrowding rates for housing in Europe at 40% of the population (the average for European countries was 17%), while in Sweden about 11% of the population were living in an overcrowded household (EU-SILC data, 2016). As to the situation of senior citizens in society, Sweden is ranked first in the Active Ageing Index (AAI) across EU member states while Latvia is among the countries at the bottom (Zaidi, 2015). Accordingly, Latvia and Sweden are markedly different regarding aging and housing issues and thus of interest to compare.

Involving Senior Citizens in Research and Innovation

Despite the fact that involving senior citizens in research and innovation in their capacity as users leads to findings with greater relevance and likelihood for implementation (Fudge, Wolfe, & McKevitt, 2007), senior citizens are seldom involved as potential users in ICT tool development (Whitney et al., 2011). This leads to an imbalance between perceptions about older people’s needs and their actual needs. Fictive users based on assumptions about senior citizens’ needs are often used instead of real users. The lack of knowledge about senior markets and marketers’ reluctance to provide unwanted associations with signs of age do not encourage improvements of design solutions, and interpretations of older people’s needs are done by professional actors in the public sector and not by senior citizens themselves (Östlund, Olander, Jonsson, & Frennert, 2015). Involving senior citizens in usability studies may be of particular value because one can expect that their life experience, including experience of functional limitations, has made them more discerning about potential flaws and also better able to judge and predict such usability issues (CEN/CENELEC Guide 6, 2002; Stephens, Carswell, & Schumacher, 2006). Usability issues are important to address because senior citizens are increasingly embracing new technology that is easy to learn, use, understand, and give potential benefits (Magnusson et al., 2002). The development of ICT tools targeting senior citizens demands an inclusive approach to ensure that the needs of the widest possible audience are addressed, taking into account the higher prevalence of functional limitations in this population segment. Age related functional limitations include reduced fine motor skills and limitations in cognition, sight, and hearing. That means the user interface has to be designed to meet particular demands in order to be approved and appreciated by this target group. A research approach embracing how senior citizens use and are affected by an ICT tool is expected to generate findings with greater relevance and create conditions for the development of a proactive tool that meets the needs and demands of today’s and tomorrow’s senior citizens.

The development and usage of different kinds of mobile applications that make senior citizens’ everyday life easier are increasing. Mobile Internet use has the potential to allow users to advocate for and meet their own needs and wishes. New technologies can gather large amounts of personalized data and compute results that can be visualized to support decisions and allow senior citizens to live independent lives longer.

There are many different approaches to conduct a usability study due to the inter-disciplinary focus and the historical development of the field of human-computer interaction (Lazar, Feng, & Hochheiser, 2010). We applied a broad approach to the usability study of an Android based mobile application (further on called the App) in a first functional stage. The entire user experience was examined, looking at the participants’ interaction with the App as well as the verbalized thoughts, feelings, and perceptions resulting from the interaction. The usability study included formative evaluation of the performance of relevant tasks, that is, identifying or diagnosing problems and making recommendations for improvement (Tullis & Albert, 2008).

Research Objective

The objective of the present study was to identify and compare elicited usability issues involving senior citizens performing tasks on an App intended to support decisions related to housing accessibility in two different national contexts: Sweden and Latvia.

Methods

The following sections discuss the context and design of the study; the participants; ethical considerations; the materials, methods, and procedures used during the study; and usability data sets and issues.

Project Context

The present study was a part of the innovAge project (www.innovage.group.shef.ac.uk/), financed by the European Commission (2013–2015). Our subproject targeted people aging with functional limitations to raise their awareness about accessibility issues in their current dwellings and in available rental apartments within the ordinary housing stock. The ambition of the ICT tool development was to support decision-making and to make it possible for users to get an overview and make predictions about housing accessibility in relation to their present or projected functional capacity. The methodological platform for the App was the Housing Enabler (HE), a research based methodology for assessment and analysis of housing accessibility problems (Iwarsson, Haak, & Slaug, 2012). The HE consists of one checklist for functional capacity in the individual and another checklist for environmental barriers indoors, at entrances, and in the immediate exterior surroundings of a dwelling. After assessment, the information collected with the two checklists is used for an analysis of person-environment fit (P-E fit), resulting in an accessibility score.

In a previous phase of the development process, senior citizens and people involved in housing construction and housing provision in Sweden, Germany, Latvia, and Italy were engaged in research circles (Härnsten, 1994; Kylberg, Haak, Ståhl, Skogh, & Iwarsson, 2015). The research circles were conducted to generate knowledge about how senior citizens express their own needs and expectations regarding housing options and to establish the user requirements for a housing accessibility ICT tool (Haak et al., 2015). The research circles produced a user requirement specification (URS) for features and content of the App. A core set of minimum requirements, common for the four countries, was identified for a universal solution with the potential to work well for a majority of users with functional limitations in different national contexts across Europe. The URS was a key component for the technological development of the App. A software professional (sixth author for this paper) was employed to develop the first fully functional prototype and give it a functional and attractive interface to be formatively evaluated for usability.

Study Design

We refer to a usability study as a process that employs people who are representative of the target audience as participants to evaluate and improve the quality of a product. The study design can be described as a problem discovery approach (Tullis & Albert, 2008) with the goal of identifying usability issues with an innovative ICT tool that had not previously been evaluated. The participants were asked to self-rate their current or future functional capacity and use information about their present housing situation to carry out the tasks. Accordingly, the tasks were specific, familiar, and relevant only to the individual participant, and the goals were influenced by their past experiences, resources, and expectations. Therefore, the data collected included elicited usability issues and self-reported data, but did not include measures of effectiveness and efficiency (e.g., task success, task time).

The present usability study concerned formative evaluation. Usability issues were identified, sorted, categorized, rated for severity, analyzed, and reported. We did not only focus on flaws, that is, usability issues that have negative connotations. We also targeted usability issues of a positive or general character, such as aspects that exceeded the participants’ expectations for usability and the appropriateness of the App. According to Tullis and Albert (2008) it is helpful to include positive usability issues to generate knowledge about design solutions that are well-functioning and appreciated. This can also help to avoid messing up positive aspects across design iterations.

One user panel in Sweden and another in Latvia evaluated the App in a Swedish and a Latvian version, respectively. Identical cross-national guidelines were developed and followed for recruitment, study procedure, and data analysis.

Participants

The participants were recruited from predefined geographical housing districts in the Skåne region in Sweden and in Riga, Latvia, including the suburbs of Riga. In both countries, potential participants were identified via contacts with hospital departments, primary health care agencies, or senior citizens/patient organizations. One participant in Sweden and two participants in Latvia were recruited from among the senior citizens who had participated in the previously completed research circles (Haak et al., 2015). Purposeful sampling (Patton, 2015) was employed to ensure that cognitively intact individuals with some practical experience of using computers, commitment to participate, willingness to share their experiences in a series of sessions, interest in issues regarding housing and health, and ability to verbalize thoughts were recruited. We strived for diversity in age, sex, civil status, computer skills, functional capacity, use of mobility devices, years of retirement, years lived with functional limitations, and type of housing. Spouses were welcomed to attend throughout the usability study procedure. However we did not collect study data on them but acknowledged their feedback regarding usability issues of the App. Accordingly, two spouses in Sweden and one in Latvia attended the sessions.

A total of 20 senior citizens (10 women, 10 men; mean age 75.4 years) participated (spouses not included). See Tables 1 and 2 for descriptive data. Participant characteristics and the description of their housing situation are based on self-reported answers collected during telephone recruitment (sample matrixes completed by the recruiters) and study sessions (questionnaires completed by moderators and participant’s self-ratings of 14 questions about functional limitations in the App). Two user panels each involving 10 participants were formed, one in Sweden (Lund) and one in Latvia (Riga).

Table 1. Participant Characteristics, N = 20

Variable	User panel in Sweden, n = 10	User panel in Latvia, n = 10	Total N = 20
Sex, n Men Women	5 5	5 5	10 10
Age group, n 65 – 74 75 – w	4 6	4 6	8 12
Civil status, n Married / living with partner Widowed / living alone	6 4	6 4	12 8
Spouse attending the sessions, n	2	1	3
Education, n No post-secondary school Post-secondary school	2 8	2 8	4 16
Functional limitations, n a Difficulty interpreting information Visual impairment Blindness (both eyes) Loss of hearing Poor balance Incoordination Limitations of stamina Difficulty in moving head Reduced function in arms Reduced fine motor skills Loss of function in arms Reduced function in spine/hips/knees Dependence on walking aids Dependence on wheelchair No functional limitations	0 3 0 5 7 3 3 3 2 1 0 5 5 0 1	1 5 0 2 4 4 2 3 4 2 0 3 2 2 0	1 8 0 7 11 7 5 6 6 3 0 8 7 2 1
Social living situation, n Single living Living with partner Living with family member(s)	4 6 0	2 5 3	6 11 3
Computer skills, n Novice Experienced user Daily user, expert	1 1 8	5 3 2	6 4 10

^a More than one functional limitation could be reported. The 14 descriptors are simplifications of the questions used in the App.

Table 2. Participants’ Housing Situation, N = 20

Variable	User panel in Sweden, n = 10	User panel in Latvia, n = 10	Total N = 20
Type of dwelling, n Apartment Single-family house	7 3	9 1	16 4
Number of rooms, n 1-2 3-4 5 or more	4 3 3	4 5 1	8 8 4
Years in current dwelling, n < 14 years 15 years or more	5 5	1 9	6 14
Housing adaptation, yes (no) n	3 (7)	1 (9)	4 (16)

Ethics

The ethical regulations in both countries were followed. According to national legislation, formal ethical approval in Sweden was granted by the regional Ethical Board in Lund (2014/573). In Latvia permission was granted by Ethical Committee of Riga Stradiņš University. In both countries, it was explicitly stated that it was voluntarily to participate. After verbal and written information about the study was given, the participants signed an informed consent form and were informed that they could withdraw from the study at any time without giving any explanation. The participants did not receive any incentives.

Material and Methods

The ICT tool was a fully functional Android based prototype App that was optimized for use on a 10-inch touch tablet, further referred to as Tablet in this paper (manufacturer: Sony Mobile Communications AB, model: Xperia^TM Z2, SGP521). The App was designed to help and guide the user to find housing solutions with the best possible accessibility. When using the App for the first time, users had to answer 14 questions about their functional capacity as a first step (see Figure 1A). The responses could either reflect the user’s current status or be a projection of a possible future situation. In the forthcoming steps, the participant’s functional capacity identified by the HE (underlying matrix; Iwarsson & Slaug, 2010) influenced the results produced by the following two main functions of the App:

• Identifying environmental barriers that (according to the HE) potentially induce accessibility problems. Using information based on the user’s functional capacity, this function identifies 0–60 environmental barriers that, if present in the current dwelling, would potentially induce accessibility problems. Descriptive texts and photos illustrating the environmental barriers were included in the App (see Figure 1B). This function was intended to be applicable when the user wishes to remain in the current dwelling and wants information on the environmental barriers that should be addressed in order to improve or maintain housing accessibility.
• Searching for a new apartment with the best possible accessibility, that is, the best match between the functional capacity of the user and the presence or absence of environmental barriers in available apartments (see Figure 1C-D). This function is applicable in situations when the user intends to move and is searching for a new apartment that meets his/her/the family’s needs and requirements in terms of accessibility. The App was connected to a database with detailed, authentic information on environmental barriers for more than 400 apartments in three municipalities in Sweden and one in Latvia, recorded by trained staff (Iwarsson, Slaug, & Malmgren Fänge, 2012). In addition, the database included information on the apartment addresses, sizes, floor locations, and monthly rents. The App allowed the user to access Google maps to see the location of the apartment. The apartments were purposefully selected to represent a wide range of building types with different standards and design.

Based on wishes elicited in the previous research circles (Haak et al., 2015), the App offered the possibility to print the results from the two main functions or save them as PDF files. To allow the participants to test the print option, a printing application (PrinterShare^TM Premium) was installed on the Tablets, and the moderators (see the Procedure section) brought portable mini printers (HP Officejet 100 Mobile Printer) to each session with participants.

Figure 1. Screenshots from the App describing A) entering functional capacity, B) list of environmental barriers identified as potentially inducing accessibility problems in current dwelling, C) search apartment, and D) list of environmental barriers in the selected dwelling according to professional assessment.

For the data collection, we used a study-specific questionnaire to gather data about the participants (see Table 1) and the participants’ housing situations (see Table 2); a pre-test rating form with five statements (Table 3); and a post-test rating form (Table 4) with statements about knowledge about accessibility, terminology, and the benefits of the App. The post-test rating form included a modified version of the System Usability Scale (SUS) with 10 statements (Brooke, 1996). The modification to the SUS was a change of the word “system” to “App.” The pre-/post-test rating forms (including SUS) had a 5-point graded scale ranging from 1 = strongly disagree to 5 = strongly agree.

SUS scores were calculated on a 0 to 100 scale (Userfocus, 2016). The SUS consists of ten statements with mixed positive and negative wording. For positively worded statements, the score contribution is the response value minus 1, and for negatively worded statements the score contribution is 5 minus the response value. Therefore, each contribution ranges from 0 to 4. The SUS is the sum of all score contributions for the ten statements multiplied by 2.5. The SUS scores ranges between 0 and 100 in a 2.5-point increment where higher values reflect higher satisfaction. A SUS score more than 68 is considered average or acceptable (Bangor, Kortum, & Miller, 2009). Descriptive statistics were used to analyze data from the SUS, pre-test, and post-test rating forms.

Table 3. Statements in the Pre-Test Rating Form

The App appears to be easy to use. I know a lot about housing accessibility. I think I will understand the terminology used. The utility of this App appears to be valuable. I value accessibility as the most important aspect in relation to other housing aspects.

Note. Each statement had a 5-point graded scale ranging from 1 = strongly disagree to 5 = strongly agree.

Table 4. Statements in the Post-Test Rating Form

· I think that I would like to use this App frequently. · I found the App unnecessarily complex. · I thought the App was easy to use. · I think that I would need the support of a technical person to be able to use this App. · I found the various functions in this App were well integrated. · I thought there was too much inconsistency in this App. · I would imagine that most people would learn to use this App very quickly. · I found the App very cumbersome to use. · I felt very confident using the App. · I needed to learn a lot of things before I could get going with this App.

· I know a lot about housing accessibility. · The utility of this App was valuable. · I understood the terminology used.

Note. The first 10 statements is the modified version of System Usability Scale (SUS) and the three last are additional statements. Each statement had a 5-point graded scale ranging from 1 = strongly disagree to 5 = strongly agree.

 

In addition, a semi-structured interview was used where the moderator asked a series of debriefing questions such as: “What did you like most/least about the App?”; “Did the usage of the App produce some thoughts?”; “What did you learn?”

Think-aloud protocols involving the participants’ verbal thoughts were used (Boren & Ramey, 2000; Nielsen, 1992; Shi & Clemmensen, 2008). Thus, the participants were describing what they were doing, what they were trying to accomplish, how confident they were about their decisions, their expectations, and why they did certain actions as they performed tasks in the App and on the Tablet. A special benefit of this technique is that it helps the moderator to observe and evaluate diverse cognitive activities in the participants (Hsieh, 2011).

Procedure

The sessions were moderated by experienced researchers in both countries. Their experiences included engaging senior citizens in research processes, carrying out research studies in home environments, and moderating group meetings and interviews with participants. The Swedish moderator (first author, PhD in industrial design) was responsible for the planning of the usability study. The procedure was developed in consultation with a usability analyst at a multi-national telecommunications equipment company and in collaboration with the Latvian moderator (third author, PhD in medical science), the software professional (sixth author) as well as the other authors (members of the interdisciplinary research team). The fact that the moderators were natives from the respective countries enhanced cultural sensitivity. In parts of the sessions, the moderators were supported by native research assistants. Throughout the study, researchers from both countries had regular face-to-face and Skype meetings for successive planning, ensuring that the sessions and analyses were carried out and followed up in a structured consistent manner. Cross-cultural understanding was enhanced by the fact that the Latvian moderator had lived in Sweden, trained in the HE methodology, and worked with the Swedish research team for two years. In addition, the Swedish researcher (first author) visited Latvia two times during the process to observe sessions.

The procedure consisted of three sessions: two individual and one user forum. Each session lasted for 1–3 hours (see Figure 2). The two moderators implemented a pilot session before the first real session in the respective country. Experiences from the pilot sessions did not lead to any fine-tuning of the procedure, but it prepared the moderators for the sessions.

Figure 2. The flow of sessions in the procedure.

First individual sessions

The first individual sessions were conducted at home visits (see Figure 2) to strive for realism and task performance that was relevant to the individual participant (Tullis & Albert, 2008). Home is also a place where the participants felt comfortable, with an empowering atmosphere where they felt encouraged to test the App. The first sessions were conducted as one-to-one sessions between the participant and the moderator (with the exception of the three sessions in which spouses attended).

The place for interacting with the Tablet that the moderator brought to the site was decided in dialogue with the participant. Seating with a table surface in front was chosen where the participant and the moderator could have their materials and sit next or opposite to each other. Firstly, the moderator asked questions to collect descriptive data (see Tables 1 and 2). The sessions continued with the moderator reading an orientation script aloud to the participant, including a short introduction to the session, a detailed explanation of the session plan, timing, and what the participant could expect of the session. Then, the participant got basic information about the interface components of the App and Tablet in a way that he or she was able to perceive, which in some cases included adjustments of display contrast and font size. Before using the App, the participant filled in the pre-test rating form to document attitudes and first impression (see Table 3). While the participant was performing tasks on the App, the moderator communicated with the participant if he/she asked for help, got stuck, or if the moderator thought it was necessary in order to understand the participant’s speech or elicit specific information. In line with the problem discovery approach applied, the user was encouraged to verbalize thoughts and share feelings when interacting with the App. If needed, the moderator helped, reminded, acknowledged, or probed questions.

Each participant independently explored the App. The App guided the participants through questions and alternatives in an interactive sequence. The participants were supposed to enter their authentic information about functional capacity, get information on environmental barriers identified as potentially inducing accessibility problems in the current dwelling, and search for and review available apartments. Table 5 describes the tasks presented to the participants. If a participant responded “NO” to every question about functional limitations, the App would generate an empty environmental barrier list, and it would show every available apartment as a perfect match. In such cases, the first individual session was stopped. Instead, the second individual session procedure was followed where the participants provided information on functional capacity based on a potential scenario with projected future loss of functional capacity, but with all questionnaires administered as in the first session. Accordingly, one participant in Sweden who did not have any functional limitations completed the individual part of the study procedure in one session. During the individual sessions the moderator observed and took field notes.

Table 5. List of Tasks in the First Individual Sessions

Enter information on functional capacity. Review list (descriptive texts and pictures) of environmental barriers identified as potentially inducing accessibility problems in current dwelling (n = 0–60). Enter criteria to search for a new apartment (geographical area, number of rooms, monthly rate, and floor). Review and choose an apartment from a list of available apartments. Print copies of the list of environmental barriers which, if present in the current dwelling, would potentially induce accessibility problems and the list of professionally assessed barriers in chosen apartment. Find a definition of housing accessibility.

If the participant did not do some of the tasks listed (Table 5) when she/he independently explored the App, the moderator asked her/him to do them as complementing tasks. When all the tasks were completed, the participant was asked to fill in the post-test rating form to determine attitudes and impressions after usage (see Table 4).

Finally, the moderator and the participant had a debriefing that included a discussion and summary of what happened during the session. Debriefing with participants is an opportunity to learn why error, difficulty, and omission occurred for every participant for every session (Rubin & Chisnell, 2008). The moderator also asked the specific debriefing questions in a semi-structured interview.

Second individual sessions

The second individual sessions were carried out 2–30 days after the first sessions and started with a discussion and summary of the experience of the first session. The subsequent parts of the second individual sessions were conducted similar to the first sessions. The only exception was that the participants provided information on functional capacity based on a potential scenario with projected future loss of functional capacity. One session in Sweden and two in Latvia were carried out privately outside the home.

User forums

The user forums (third sessions) sought to provide space for dialogue among users about the usability and appropriateness of the App. The user forum was a method to gather user panel participants to discuss common usability issues, listen to one another, have the time to process what others say, and enrich each other’s thinking without requiring them to reach a consensus. All 10 participants and the participating spouses were invited to the user forum in the respective country. Accordingly, six participants and two spouses attended the user forum in Lund and six participants attended the user forum in Riga. Each user forum was facilitated by the moderator and the native assistant researcher at the respective research site. The Swedish moderator took part in the user forum in Latvia and gave a presentation in English that was simultaneously translated into Latvian by the native moderator. The user forums included the following four steps:

1. presentation of a summary of the preliminary results from the first and second sessions from both countries including participants’ feedback
2. discussion on usability issues in small groups (2–3 participants in each) with App interaction followed by presentations by the participants of the group discussions for all user forum participants
3. presentation of possible solutions and improvements of the App including participants’ feedback
4. summary of the session led by the native assistant researcher participating in the user forum including participants’ feedback

During the user forums, the researchers encouraged the participants to discuss and share their experiences with each other and with the researchers, ensuring that all participants had a chance to speak. The user forums lasted for 3 hours including a coffee break.

The Usability Data Set

In the individual sessions, the data was collected in the form of completed questionnaires, rating forms, and field notes taken by the moderator during each individual session. The field notes included think-aloud comments from the participants, the observations made by the moderator, and the participants’ answers to the debriefing questions. In the user forums, the data was generated in the form of field notes and audio recordings.

Usability Issues: Analysis of Data From All Sessions

Field notes and audio recordings were analyzed separately in each country and then compared cross-nationally. Usability issues were identified using a pragmatic approach inspired by manifest content analysis (Graneheim & Lundman, 2004). Content analysis is a systematic description of written, spoken, or visual data. A pragmatic scientific approach to analysis is relevant, especially in applied research (Patton, 2015).

Field notes from the first and second sessions were read several times and relevant notes about usability issues were identified and listed. The usability issues identified in the sessions in Latvia were translated to English. The first author then sorted the usability issues into the interaction sequences: enter information on functional ability, review list of potential barrier in current dwelling, enter criteria to search for a new apartment, review list of available apartments, and make printouts of the lists and information/help. There was also a general category not related to a particular interaction sequence. The usability issues were also divided by the first author into categories labeled flaws (issues that caused problems for the participants and at least gave a hint of which part of the interface the developers needed to address) and other issues (positive and general comments and observations on aspects of use that did not directly point to possible improvements but could be important to consider in the formative evaluation and development process). In the next step a priori coding approach (Lazar et al., 2010) was taken where the usability issues were categorized separately by the first and last author according to type: content, functionality, navigation, or terminology. Flaws were categorized according to the severity: irritant, moderate, severe, or unusable (Tullis & Albert, 2008; Rubin & Chisnell, 2008). After the individual categorization, the first and last author met, discussed, and reached consensus.

Based on the categorized usability issues generated from the first and second sessions, a preliminary summary was made by the moderator in each country. It included a description of the most significant usability issues, why they were issues, and the severity of the flaws. Positive and general comments and observations were also included. In Sweden, possible solutions were included in the preliminary summary because the Swedish moderator was part of the design team. The preliminary summaries were used to develop possible solutions and improvements of the App to present in user forums.

After the completion of the user forums, the field notes were read and the audio recordings were listened to several times. Relevant usability issues were identified, translated to English (only the usability issues identified in Latvia), listed, sorted, and categorized in the same way as the usability issues generated in the first and second sessions. The purpose was to enter all the data into a form that allowed the researchers to reveal patterns and understand which flaws were the most important to address. A final report was written and used as a base for further development of the App.

Results

The following sections present the pre- and post-test rating forms; the SUS scores; and the usability issues according to type: content, functionality, navigation, and terminology.

Pre- and Post-Test Rating Forms

The results from the pre-test rating form indicate that participants in Sweden had higher expectations of the usability of the App than the participants in Latvia. The participants in Latvia rated the importance of housing accessibility higher than the participants in Sweden.

The pre- and post-test rating comparison showed that after usage of the App the participants in Sweden rated that they had learned more about housing accessibility, which was not the case for the participants in Latvia. In both countries, ratings of the benefits of the App and the understanding of the terminology increased after usage.

SUS Scores

The participants in Sweden scored on average 76.3 in the first session and 85.0 in the second session on the SUS scale, compared to the participants in Latvia who scored 68.8 in the first session and 67.5 in the second session. That is, the usability of the App was considered at least average or acceptable by both panels though the participants in Sweden scored higher on the SUS scale than the participants in Latvia. The SUS scores by the Swedish participants also indicated an increased satisfaction from the first to the second session.

Usability Issues

In total 672 notes (321 elicited with participants in Sweden, 351 in Latvia) were identified as expressing usability issues. Most of them, both flaws and other issues, concerned functionality (Figure 3). Flaws that were categorized as severe or unusable were also mostly related to functionality. However, compared to the usability issues elicited with participants in Latvia, the usability issues elicited with participants in Sweden were more about content, but the majority of these issues were categorized as moderate. Overall a higher number of flaws were elicited with participants in Latvia compared to the number of flaws elicited with participants in Sweden (296 vs. 218 respectively). With regard to interaction sequence, printouts and problems of a general character generated relatively high numbers of notes categorized as flaws by the participants in Latvia (Figure 4).

Figure 3. Number of flaws (n = 514) and number of other issues, positive and general comments or observations (n = 158), categorized by type and severity (in case of flaws).

Figure 4. Number of flaws (n = 514) by interaction sequence, help/information, and a general category not related to a particular sequence of interaction.

Identified Usability Issues in Summary

For a summary of usability issues identified in the two countries, see Table 6.

Table 6. Summary of Usability Issues by Issue Type

Issue type	Swe	Lat	Examples of identified usability issues
Content	X	X	Lack of information on how to address accessibility problems.
	X	X	The barrier illustrations were perceived as unclear and ambiguous.
		X	Some barrier descriptions/illustrations were perceived as irrelevant.
	X	X	The accessibility scores and ranking of the available apartments were not noticed, and if they were noticed, they were perceived as difficult to understand.
	X	X	Lack of information about features that make an available apartment accessible.
	X		Lack of vital information about the available apartments.
	X		Wishes for more graphs, photos, and colors for a more helpful and pleasurable experience.
Functionality	X	X	Lack of possibility to interact with the App and checkmark whether a listed barrier was present or not in current dwelling.
	X	X	Lack of overview of which items had been reviewed in the list of barriers.
		X	Appreciation of the possibility to “play” with the apartment search function to check out and compare with options available in other countries.
		X	The printing process was perceived as troublesome and sometimes unnecessary.
	X		The possibility to explore accessibility aspects that could emerge when looking for a new housing alternative was appreciated.
	X		Wishes for a search function.
	X	X	Too low contrast and too small font size for people with visual impairment.
		X	Wishes to use the ICT tool from a PC or laptop.
Navigation	X	X	Unclear what to achieve by choosing between the two main functions.
	X	X	Difficulties with back navigation.
		X	Severe usability flaws related to navigation by use of a touchscreen.
Terminology		X	The expressions/descriptors of the YES/NO questions on functional capacity were considered to be too long and difficult to understand.
	X	X	The language used in the App was not perceived as appropriate for the targeted audience.

Identified Usability Issues Concerning Content

Many notes that expressed flaws of the App concerned the barrier illustrations. The photos were―in many instances―perceived as unclear, ambiguous, or irrelevant in relation to the situation of the participant. Particularly among participants in Latvia, some parts of the list of environmental barriers were considered dependent on a certain national context.

Some participants expressed that they liked the list of barriers that was perceived as excellent, detailed, useful, and worth devoting time and attention to acquiring knowledge. One comment was that “There are many barriers that I haven’t really noticed” (participant in Sweden).

Common for participants in both countries was a wish to get information on how to act on environmental barriers that turned out to be present in their current dwelling, such as do-it-yourself tips, photos illustrating good examples, or information about whom to contact to get financial or practical support. Without such content, the App had no benefit according to some participants. Moreover, the participants wished for content that informed on how to act in order to influence the municipality to work towards more accessible and age-friendly environments.

Participants in both countries had difficulties understanding the accessibility scores generated by the HE (underlying matrix) and ranking of the apartments that were presented in the list of apartment search results. The participants in Sweden expressed more clearly that they perceived lack of content when searching for a new apartment than the participants in Latvia.

The map feature was appreciated because it created possibilities to search for, check out, and compare apartments in different housing areas.

Participants in both countries asked for information on features that make an available apartment accessible (i.e., positive features) and not only the environmental barriers (i.e., negative features). A participant from Sweden commented, “It’s logical to only list the environmental barriers, but it’s an inadequacy that isn’t satisfactory.” Additional similarities with participants in both countries were lack of information about (a) distances to service and public transport from the presented apartment and (b) whether the existing environmental barriers were easy to resolve or not.

Many of the usability issues raised in both countries concerned content beyond housing accessibility (i.e., beyond the scope of the App). Content desired by the participants when reviewing apartment search results was information similar to what a real estate broker would provide.

Identified Usability Issues Concerning Functionality

In both countries, participants with multiple functional limitations did not see how they could benefit from the functionality to change the information about functional capacity and use the App to forecast a projected future (i.e., a task in second session). Reasons were that they already had received a list with nearly all of the 60 possible environmental barriers and the fact that their personal experiences of living with functional limitations already made them aware of the accessibility problems in their dwelling. In contrast, the option to use the App to consider a possible future situation was appreciated by those who did not currently have accessibility problems. One comment was that “People should reflect on their possible need before it is too late” (participant in Sweden).

In the list of environmental barriers identified as potentially inducing accessibility problems, participants in both countries lacked two major functionalities. First, the participants wanted to be able to checkmark whether the listed environmental barriers were present or not in their current dwelling. Because it was not possible to give input to the App, many participants perceived that it was unclear what they should do with the list of environmental barriers generated. Several participants actively wanted to give input such as information on their own perceived accessibility problems and describe their own housing situation. Secondly, the participants lacked an overview of which barriers they had reviewed.

Participants in both countries experienced flaws regarding the printing functionality. Flaws arose chiefly because the printing procedure required many operational steps. Technical problems also occurred with the portable mini printer and its connection to the printing application. The printing process was perceived as more troublesome for the participants in Latvia than for those in Sweden. Examples of flaws encountered were that the texts in the printing application were only available in English, and the font size was too small to be legible. However, the printing option was generally appreciated in both countries.

Some flaws were related to the functional abilities of the participant. For example, participants with visual impairment commented on too low of a contrast and too small of a font size. The possibility to get verbal feedback when using the App was also asked for.

 

 

Identified Usability Issues Concerning Navigation

In both countries, many of the participants perceived that it was unclear what they should achieve by using the App, and also the difference between the two main functions was unclear.

Many of the flaws concerning navigation were related to the practical handling of a touchscreen. These issues arose in both countries, but were more frequently encountered among the participants in Latvia. Participants with no previous experience of touchscreens and people with reduced fine motor skills had difficulties carrying out the tasks on their own and exploring all functionalities.

The fact that the App was in a first fully functional prototype stage and occasionally crashed when the participants navigated between different functions also led to additional trouble.

Identified Usability Issues Concerning Terminology

Several of the participants in Latvia perceived that the expressions/descriptors of the “YES” or “NO” questions on their functional capacity were too long and difficult to understand (e.g., Figure 1A). These flaws made the participants in Latvia uncertain as to how to answer these 14 questions, and they wanted the option to give nuanced responses to a higher degree than the participants in Sweden. Still, the majority of the participants in Sweden and some in Latvia had no problems answering the questions and considered it as valuable to evaluate their own functional capacity in a structured way.

Participants in both countries stated that the terminology used was difficult to understand (e.g., environmental component, functional profile, resting surfaces, and necessary dwelling functions). The environmental barrier descriptions were experienced as problematic, especially in the list of available apartments where no further explanations were given nor photos shown. This led to hesitation, additional needs of help, and more guidance than provided by the App, which consequently led to a lack of motivation.

Discussion

Based on user involvement from the outset of the research process, the main contribution of the present study is that it adds knowledge and insights on the kind of usability issues that arise when an innovative ICT tool is evaluated and developed in a cross-national setting (Sweden and Latvia) and for a relatively unexplored target group, that is people aging with functional limitations. The high number and the severity of usability issues related to flaws in functionality point to a need for a more thorough exploration of design solutions that are tailored for this target group. Moreover, many of the content related issues highlight how national and cultural contexts may influence perceived usability and acceptance of a new ICT tool in a seemingly coherent target group, and therefore indicate the need to consider these early on and throughout the development process.

We identified and compared usability issues of an App with a socially relevant ambition: to address housing accessibility problems among senior citizens. Overall, the participants in both countries appreciated the ambition to produce personalized information that can be used to support decisions on housing accessibility. The human-computer interaction literature (see for example Lazar et al., 2010) does not generally recommend the utilization of a participant’s personal data in usability studies but rather data that are fabricated for the study. However, like Genov, Keavney, and Zazelenchuk (2009), we found that the usage of authentic information and tasks related to the participants’ own situation resulted in realistic studies with engaging data. In addition, it helped the participants to understand the purpose and possible benefits of the App.

Though there was some initial hesitation and questioning particularly among the Latvian participants, the structured evaluation of the functional capacity and the possibility to learn about how environmental barriers can generate problems in relation to different functional limitations was highly appreciated. In addition, the participants in Sweden found it interesting to explore the accessibility aspects that could emerge when looking for a new housing alternative. The usefulness of the App as an awareness raiser about housing accessibility issues was further illuminated by the reactions of the participants to the list of barriers potentially generating problems in current or future dwellings. Many participants, both in Sweden and Latvia, were surprised by the length and detail of information provided, and they acknowledged that it included many aspects they had not previously considered. Recent research also indicated that even though senior citizens who are relocating recognize the importance of moving to a dwelling with fewer environmental barriers, they seldom sufficiently consider the possible consequences of further deterioration in functional abilities (Granbom et al., 2014). The use of a tool such as this App could therefore be relevant to consider also for housing market professionals that are counselling senior citizens and sell or rent out dwellings.

While the characteristics of the participants in the two countries were largely similar, there were some differences to take into consideration when interpreting the results. None of the participants in Sweden lived with family members other than their spouse, while three out of ten of the Latvian participants did. Living with younger family members may reduce the need to address accessibility problems, as people in the household can support each other. Such situations also imply that the individual is not alone in the decision-making about the housing situation. Another difference that may influence the Latvian participants focus on how the App could support them to improve their current dwelling was that the participants in Latvia had lived in their present dwelling for a longer period of time compared to the participants in Sweden. Generally, people living in a home for a longer time prefer to stay put and are more reluctant to move (Abramsson, 2009). Yet other reasons might be that a majority of senior citizens in Latvia are limited in financial possibilities and the choice of affordable apartments is very limited. Any ICT tool, such as the App evaluated in this study, needs to fit the targeted national and cultural context. One possible way could be to allow easy adjustments to local, regional, and national needs.

Another difference to note was that the participants in Sweden had more experience in using computers and touchscreens. The novice level of touchscreen skills among participants in Latvia made them more hesitant and more focused on the functionality of the App compared to the more touchscreen-literate participants in Sweden. This kind of finding underlines the importance of reflecting on strategies to overcome national and cultural differences in future projects. With the objective to develop an ICT tool for use in a European context, it would be most valuable to involve a broader range of expertise during the development phases (Bannon & Ehn, 2012). In this way, challenging issues in a cross-national context could be considered from the start and pave the way for a more effective and ecologically valid development.

Implications for Future Development

Even though overall satisfaction of the App was acceptable on the SUS scale, comments from the participants indicated there was need for further development and additional features to enhance the potential benefits. For example, it was suggested future development should include interactive features that allow users to provide input on their own situation such as own photos or descriptions. Some participants recommended a user discussion forum enabling senior citizens and informal caregivers to communicate, share their experience, and support each other. It could even provide the possibility to contact and give input directly to relevant stakeholders such as municipal officials or urban planners. Development of such future features would enable empowerment and create possibilities to influence physical planning towards more age-friendly environments. Moreover, that would be in line with the long term goal of the App, which is to actively involve senior citizens in housing provision as critical consumers and ultimately influence housing policies and provision practices across Europe. Such future developments could also be part of an adaption to the changing conditions in the production of physical environments in Europe (Bannon & Ehn, 2012; Sieverts, 2013) and encourage new ways for self-organized planning and making age-friendly environments.

Furthermore, the feedback provided by the user panels contributed to a deeper understanding of the many challenges that lie ahead if the App is to be applicable on the European market. For example, the housing provision situation in different countries has to be considered, just as the differences in housing standards, legislation, and systems for financing housing adaptations.

It should further be remembered that during the usability study, the apartment database used by the prototype App was static, but for a released application, a dynamic database with available apartments for rent (see Figure 1C-D) will be necessary. Future work includes developing, testing, and evaluating a new system for inventorying environmental barriers in available apartments in multi-family housing as a basis for improved information to different groups of housing applicants, on the options available in the local housing market, and about how accessible different parts of the housing stock are.

Methodological Considerations

The moderators’ different backgrounds may have influenced their understanding of what usability issues are and thereby influenced the findings. Different disciplinary perspectives have to be considered because there may be subtle forms of biases. Moderators may pay attention to details most relevant to their discipline, often in ways that they are unaware of. According to Lazar et al. (2010), “By necessity, we filter what we see and hear, and interpret our observations through the lenses of our own history, experience, expertise, and bias” (p. 234). To counter such risks, the common and detailed guidelines for the moderators used in the present study were instrumental. Moreover, the study situation needs to be taken into account. For example, how the power balance between the participants and the moderator is perceived may impact on honesty of responses elicited. In this study, we chose to perform two individual sessions in the participant’s home, and our impression is that this made the participants feel more comfortable and open with expressing critical comments.

The individual sessions also gave a close connection to the users’ environments and were useful in the many aspects in understanding the App usage. Previous research has emphasized that the usage of ICT tools is related to the concrete environment and situation from which actions take their meaning (Hsieh, 2011; Millen, 2000). For example, the moderators enhanced their understanding by observing the participants interact with the App first-hand. A further reason to carry out the individual sessions in participants’ homes was that we thought it would be more convenient for the participants with functional limitations to avoid transporting themselves to a research site. According to Lazar et al. (2010) visiting participants in their home may help in recruiting them. The ambition to avoid the transportation challenge was appreciated by the participants in Sweden, but in the Latvian context, the participants were initially doubtful and hesitated to let a researcher into their home which instead made the recruitment process more complicated. However, if complications are possible to overcome with cultural and social sensitiveness, our recommendation is to use the home environment for these kinds of usability studies.

While five participants is an accepted number to identify approximately 80% of usability problems in an interface (Nielsen, 2012; Virzi, 1992), the present study involved 20 participants. Considering the complexity of functional capacity (Slaug, Schilling, Iwarsson, & Carlsson, 2011) and the diversity of interests, attitudes, and ways of living in old age (Eriksson, 2010), the sample still did not cover the heterogeneity of senior citizens. Accordingly, we recommend a higher number of participants. Moreover, the fact that the experience of using tablets/touchscreens varied between the participants in the two countries calls for more sensitive inclusion criteria concerning basic knowledge on how to interact with tablets/touchscreens.

The think-aloud protocol used in the present study is a common method in usability studies that is suitable to find out why problems occur and how participants try to work around them (Boren & Ramey, 2000; Nielsen, 1992; Shi & Clemmensen, 2008). It may also reveal important clues about how participants are thinking about the product under development. However, it is important to be aware of national and cultural differences in how comfortable a participant may feel about speaking aloud and perhaps criticizing the product. People from some cultures may not feel comfortable expressing their concerns immediately (Shi & Clemmensen, 2008). As previous research projects in Latvia gave some reason for concern in this regard (Tomsone, Zalkalns, Nygren, & Iwarsson, 2007), the moderator in Latvia made additional efforts to gain the participants confidence to talk freely. The large number of usability issues voiced by the participants in Latvia, indicates that they did not refrain from expressing critique.

As the prototype evaluated in the present study was a fully functional App, the participants might have been hesitant to criticize it. That is, prototypes that the users experience as already finished might evoke the feeling that their feedback will not matter (Lazar et al., 2010). However, we did not experience hesitation from the participants; instead they shared their critiques in various ways. The fact that the prototype was fully functional may rather have elicited usability issues of a more concrete and detailed nature such as unclear illustrations and results produced by the functions of the App, technical problems, and illogical navigation that would not have been possible with a prototype at an earlier development stage. This strategy is consistent with the findings from Hertzum (2016) who suggested moving usability evaluations to later design phases because testing too early in artificial environments may prevent the discovery of important contextual and sociotechnical mismatches.

Another aspect that should be considered is the fact that the App was evaluated in two different national versions including language translation that might have impacted the results concerning the terminology. However, the relatively low number of terminology issues identified―even fewer by participants in Latvia compared to participants in Sweden―suggests that it did not have such an impact in the present study.

Study Limitations

There was a time constraint for the study that was not satisfactory. In both countries, participants with similar characteristics were recruited and scheduled within a certain time frame. Participants with functional limitations might need extra time during the procedure as well as a time-space for rescheduling in case of health problems. Moreover, individual sessions at home visits result in substantial travel time for the researchers. However, time and resources for usability studies are mostly limited, and the rich data material we obtained do not suggest any significant negative impact for the present study.

When comparing usability data from different countries there is a risk for generalizing about the national groups of participants when in fact there are many individual differences within the national groups. Accordingly, each individual is drawing attention to different aspects of use based on his or her own previous experiences, resources, and expectations. This should be kept in mind when interpreting the differences found after the comparisons of the results of the two national user panels.

Conclusions

The results of the present study add knowledge and insights on usability issues related to an innovative ICT tool that was developed and evaluated in a cross-national setting and for a relatively unexplored target group, that is, people aging with functional limitations. The usability issues identified point to a need for a more thorough exploration of design solutions that are tailored for this target group. The results also highlight how national and cultural contexts may influence perceived usability and acceptance of a new ICT tool in a seemingly coherent target group, and thereby indicate the need to consider these early in and throughout the development process.

The usability issues that emerged during the study were useful in providing design recommendations that were subsequently implemented in the prototype housing accessibility App. The appreciation of the App that was evaluated suggests it has the potential to be accepted and used as a tool to raise the awareness about accessibility issues and to support decisions on appropriate housing solutions. The study also shows that further development and additional features will be needed to enhance the benefits and prepare the App for market release.

Tips for Usability Practitioners

The following are suggestions for usability practitioners:

• Prepare detailed guidelines in cross-national, multi-language research projects to ensure that the recruitment, procedure, and analyses are performed in the same way by all moderators in all participating countries.
• Involve individuals with diverse functional capacity to elicit a broader range of usability issues if the target group is people aging with functional limitations.
• Keep in mind that cultural aspects influence participants’ understanding of, attitudes towards, and behavior towards both participation in usability studies and ICT products. Experienced moderators with cultural knowledge and sensitivity may ensure such an informed approach.
• Carry out usability studies in real sites to reach close and first-hand connection to the participants’ environments to achieve as realistic study situation as possible. The participants’ own homes may help them to feel more comfortable and be more open with expressing critical comments. Home visits are more convenient for participants with functional limitations, as transportation to a research site will not be necessary.
• Let the participants use their personal data to create a realistic, meaningful, and engaging study situation, following guidelines on ethics and confidential data management.

Conflict of Interest

Susanne Iwarsson and Björn Slaug are the copyright holders and owners of the Housing Enabler (HE) assessment tool and software, provided as commercial products. The other authors have no competing interests.

Acknowledgements

The innovAge project was part of Social Innovations Promoting Active and Healthy Ageing, HEALTH.2012.3.2-3 No. 306058 within the Seventh EU Framework Programme. Additional funding was provided by the Ribbingska Foundation in Lund, Sweden, and was accomplished within the context of the Centre for Ageing and Supportive Environments (CASE) at Lund University, financed by the Swedish Research Council for Health, Working Life and Welfare (Forte). The authors would like to thank all the user panel participants, consortium, and team members for their contributions. The authors are specifically grateful for the valuable contributions of Ingrid Hilborn, Monica Holmgren, Zane Liepina, Liene Saukuma, Anna Schömer, and Erik Skogh.

References

Abramsson, M. (2009). Mellanboendeformer för äldre: av intresse såväl för de äldre som för bostadsföretagen. [In-between housing for seniors: Of interest both for old people and for housing companies]. In H. Jönsson (Ed.), Åldrande, åldersordning, ålderism [Aging, order of age, ageism] (Studies no 2009:10, pp. 138–156). Linköping, Sweden: Linköping University Interdisciplinary.

Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123.

Bannon, L. J., & Ehn, P. (2012). Design: Design matters in participatory design. In J. Simonsen & T. Robertson (Eds.), Routledge international handbook of participatory design (pp. 37–63). London: Routledge.

Boren, M. T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261–278.

Brooke, J. (1996). SUS – A quick and dirty usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor & Francis.

CEN/CENELEC Guide 6 (2002). Guidelines for standards developers to address the needs of older persons and persons with disabilities (1^st ed.). Retrieved May 2016 from ftp://ftp.cencenelec.eu/CENELEC/Guides/CENCLC/6_CENCLCGuide6.pdf.

Eriksson, B. G. (2010). Studying Ageing: Experiences, description, variation, prediction and explanation. (Doctoral dissertation). Gothenburg University, Sweden.

EU-SILC data (2016). Retrieved October 2016 from http://ec.europa.eu/eurostat/tgm/refreshTableAction.do?tab=table&plugin=1&pcode=tessi171&language=en.

Fudge, N., Wolfe, C. D. A., & McKevitt, C. (2007). Involving older people in health research. Age and Ageing, 36, 492–500.

Genov, A., Keavney, M., & Zazelenchuk, T. (2009). Usability testing with real data. Journal of Usability Studies, 4(2), 85–92.

Granbom, M., Himmelsbach, I., Haak, M., Löfqvist, C., Oswald, F., & Iwarsson, S. (2014). Residential normalcy and environmental experience in very old age. Changes in residential reasoning over time. Journal of Aging Studies, 29, 9–19.

Granbom, M., Iwarsson, S., Kylberg, M., Pettersson, C., & Slaug, B. (2016). A public health perspective to environmental barriers and accessibility problems for senior citizens living in ordinary housing. BMC Public Health, 16(1), 772.

Graneheim, H., & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112.

Haak, M., Fänge, A., Iwarsson, S., & Dahlin-Ivanoff, S. (2007). Home as a signification of independence and autonomy – Experiences among very old Swedish people. Scandinavian Journal of Occupational Therapy, 14, 16–24.

Haak, M., Slaug, B., Oswald, F., Schmidt, S. M., Rimland, J. M., Tomsone, S., Ladö, T., Svensson, T., & Iwarsson, S. (2015). Cross-national user priorities for housing provision and accessibility – Findings from the European innovAge project. International Journal of Environmental Research and Public Health, 12(3), 2670–2686.

Hertzum, M. (2016). Usability testing: Too early? Too much talking? Too many problems? Journal of Usability Studies, 11(3), 83–88.

Hsieh, H. C. L. (2011). Exploring the impact of cultures on web usability test. In M. Kurosu (Ed.), Human centered design (HCII 2011, LNCS 6776, pp. 47–54). Berlin Heidelberg, Germany: Springer-Verlag.

Härnsten, G. (1994). The research circle: Building knowledge on equal terms. Stockholm, Sweden: Swedish trade union confederation.

Iwarsson, S., Haak M., & Slaug, B. (2012). Current developments of the Housing Enabler methodology. British Journal of Occupational Therapy, 75, 517–521.

Iwarsson, S., & Slaug, B. (2010). Housing Enabler – Metodik för bedömning och analys av tillgänglighetsproblem i boendet. Manual för fullständigt instrument och screeningverktyg [Housing Enabler – A method for rating and analyzing accessibility problems in housing. Manual for the complete instrument and screening tool] (2^nd ed.). Lund & Staffanstorp, Sweden: Veten & Skapen HB and Slaug Enabling Development.

Iwarsson, S., Slaug, B., & Malmgren Fänge, A. (2012). The Housing Enabler screening tool: Feasibility and interrater agreement in a real estate company practice context. Journal of Applied Gerontology, 31(5), 641–660.

Kylberg, M., Haak, M., Ståhl, A., Skogh, E., & Iwarsson, S. (2015). Research brief #5: User participation. Research with and about user participation. Stockholm, Sweden: Forte [Swedish Research Council for Health, Working Life and Welfare].

Lazar, J., Feng, J. H., & Hochheiser, H. (2010). Research methods in human-computer interaction. West Sussex, UK: Wiley.

Living conditions in Europe (2014). Eurostat statistical books. Retrieved May 2016 from http://ec.europa.eu/eurostat/documents/3217494/6303711/KS-DZ-14-001-EN-N.pdf/d867b24b-da98-427d-bca2-d8bc212ff7a8.

Magnusson, L., Hanson, E., Brito, L., Berthold H., Chambers. M., & Daly, T. (2002). Supporting family carers through the use of information technology — the EU project ACTION. International Journal of Nursing Studies, 39(4), 369–81.

Millen, D. R. (2000). Rapid ethnography: Time deepening strategies for HCI field research. In Proceedings of the 3rd conference on designing interactive systems: Processes, practices, methods, and techniques (pp. 280–286), New York, NY: ACM.

Newman, A. B., & Cauley, J. A. (Eds.) (2012). The epidemiology of ageing. Dordrecht, Germany: Springer.

Nielsen, J. (1992). The usability engineering life cycle. Computer, 25(3), 12–22.

Nielsen, J. (2012). How many test users in a usability study? Retrieved May 2016 from http://www.useit.com/alertbox/number-of-test-users.html.

Östlund, B., Olander, E., Jonsson, O., & Frennert, S. (2015). STS-inspired design to meet the challenges of modern aging. Welfare technology as a tool to promote user driven innovations or another way to keep older users hostage? Technological Forecasting & Social Change, 93, 82–90.

Patton, M. Q. (2015). Qualitative evaluation and research methods (4^th ed.). Thousand Oaks, CA, USA: Sage Publications.

Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2^nd ed.). Indiana, Indianapolis: Wiley Publishing, Inc.

Shi, Q., & Clemmensen, T. (2008). Communication patterns and usability problem finding in cross-cultural thinking aloud usability testing. In Proceedings of CHI ’08 Extended abstracts on human factors in computing systems (pp. 2811–2816). New York, NY: ACM.

Sieverts, T. (2013). The adaptable city 2. Retrieved May 2016 from
https://www.arkitekt.se/wp-content/uploads/sites/7/2014/06/E13Theme-Brochure_EN-LD.pdf.

Slaug, B., Schilling, O., Iwarsson, S., & Carlsson, G. (2011). Defining profiles of functional limitations in groups of older persons: How and why? Journal of Aging and Health, 23(3), 578–604.

SOU (Statens Offentliga Utredningar) [Swedish Government Official Reports] (2015:85). Bostäder att bo kvar i: Bygg för gemenskap i tillgänglighetssmarta boendemiljöer [Dwellings to stay in: Build for social inclusion in accessible smart living environments]. Report of the investigation into housing for older people. Stockholm, Sweden.

Stephens, E. C., Carswell C. M., & Schumacher, M. M. (2006). Evidence for an elders’ advantage in the naive product usability judgments of older and younger adults. Human Factors, 48(3), 422–433.

The National Board of Housing, Building and Planning [Boverket] (2015). The housing adaption grants [Bostadsanpassningsbidragen] 2014. Report 2015:27. Retrieved May 2016 from http://www.boverket.se/sv/om-boverket/publicerat-av-boverket/publikationer/2015/bostadsanpassningsbidragen-2014/.

Tomsone, S., Zalkalns, J., Nygren, C., & Iwarsson, S. (2007). The meaning of everyday performance at home among old people in Latvia. RSU Zinātnisko rakstu krājums, 210–217.

Tullis, T., & Albert, B. (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Burlington, MA, US: Morgan Kaufmann Publishers.

Userfocus (2016). Measuring usability with the system usability scale (SUS). Retrieved May 2016 from http://www.userfocus.co.uk/articles/measuring-usability-with-the-SUS.html.

Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors: The Journal of the Human Factors and Ergonomics Society, 34(4), 457–468.

Whitney, G., Keith, S., Bühler, C., Hewer, S., Lhotska, L., Miesenberger, K., Sandnes, F. E., Stephanidis, C., & Velasco, C. A. (2011). Twenty five years of training and education in ICT Design for All and Assistive Technology. Technology and Disability, 23(3), 163–170.

Zaidi, A. (2015). Active ageing index: A legacy of the European year 2012 for active ageing and solidarity between generations. Retrieved May 2016 from http://www.euro.centre.org/detail.php?xml_id=2489.

Published in: in Volume 12, Issue 1,