The Roles of Health Literacy, Numeracy, and Graph Literacy on the Usability of the VA’s Personal Health Record by Veterans

Abstract

Personal Health Records (PHRs) that are tethered to electronic medical health systems are applications that can significantly enhance patients’ health and health care. The primary aim of this research was to examine the roles of health literacy, numeracy ability, and graph literacy in enabling a group of veterans to perform health-management tasks using My HealtheVet (MHV), the Department of Veterans Affairs’ PHR portal. Forty participants, all users of MHV, were recruited in two age groups: < 65 and ≥ 65 years of age. They were asked to perform 13 tasks representative of this portal’s eight major categories of functions, and were categorized into lower and higher performers based on their performance of those tasks. The results indicated that age, health literacy, numeracy, and graph literacy all significantly differentiated lower from higher task performers. Also, older veterans performed more poorly than their younger counterparts. Graph literacy explained a significant amount of the variability in task performance even after computer and Internet proficiency, health status, health literacy, and numeracy ability were taken into consideration. Exit interviews emphasized problems with the presentation of excessive information and with navigating this portal. Participants also offered a number of recommendations for improving this PHR’s design. Overall, the findings provided the basis for recommendations that consider both more conventional interface design issues as well as problems that could stem from individual factors such as health and graph literacy. The findings from this study are expected to inform upcoming redesign efforts of MHV as well as other relatively complex PHRs.

Tips for Usability Practitioners

The following tips can help practitioners who plan to undertake similar studies:

• When studying complex applications such as PHRs that are tethered to electronic health record systems, it is important to assess older users as a separate subpopulation. The demands imposed by the large amounts of information these types of systems must necessarily make available in numerous types of formats and the added functions for managing this information require that age-related declines in cognitive abilities be taken into account in any proposed interface design strategies or improvements, with the understanding that accommodating older users is likely to improve the usability for other users as well.
• When assessing a website, online application, or portal that provides substantial information to users, especially in numerous formats, consider the skills that users need to comprehend that information, develop measures of those skills, and use them to enhance the richness of your evaluation of the user experience. In some cases, standardized instruments may exist for capturing those measures.

Introduction

Personal Health Records (PHRs) that are tethered to the electronic medical records of patients’ health care providers are computer applications that provide a central repository of health-related information, along with a variety of tools designed to help patients see, review, share, and manage this information in a secure and confidential online environment in order to optimize their health care (Archer, Fevrier-Thomas, Lokker, McKibbon, & Straus, 2011; Tang, Ash, Bates, Overhage, & Sands, 2006; Wynia & Dunn, 2010). Although the adoption of PHRs by the public has been relatively slow, survey studies indicate that interest in PHRs is increasing, with the percentage of Americans who have reported using electronic PHRs rising from 3% to 10% between 2008 and 2011 (Markle Foundation, 2011). Moreover, PHR use is expected to increase as more physicians implement electronic health records that can interface with patient PHRs (Tenforde, Jain, & Hickner, 2011).

Use of PHRs can provide a number of potential benefits to patients. For example, the enhanced knowledge about one’s health afforded by PHRs can help patients formulate more insightful questions to their medical providers and consider a greater array of options for improving their health (Undem, 2010). Use of a PHR can also lead to decreased health care utilization or improved chronic disease control through better care coordination, access to care, communication, and patient empowerment (Ralston et al., 2009). PHRs may be particularly useful for older patients due to the increased occurrence of multiple chronic conditions and need for medical care among older adults (Administration on Aging, 2011).

Review of Related Research

To date, there have been few published studies involving the assessment of user interactions with PHRs that are designed to be tethered to electronic medical record systems. Most PHR usability studies have instead relied largely or exclusively on survey, questionnaire, or interview data. In this section, we review several relatively recent performance or scenario-based studies that addressed the usability of such PHRs through the application of both qualitative and quantitative methods.

The study by Britto et al. (2009) targeted a prototype of a web-based PHR known as MyCare Connection. Through disease-specific tabs, parents could access three different patient portals within this application that were each customized for a particular chronic disease. Sixteen participants (mean age of 39), who were parents of children being seen at a clinic of this hospital dedicated to treating one of these three diseases and who had never enrolled in any of this application’s portals, were allocated to one of the three portals specific to their child’s disease and asked to complete 14 tasks. There were three “rounds” of testing, with each round targeting one of the three subgroups of participants; thus, sequential usability testing of each portal was not performed. Following each round of testing, adjustments were made (e.g., renaming of tabs to improve navigation) to the particular portal that participants were to use based on data accrued from the participants who used the other portal(s) in previous rounds. Participants were asked to “think aloud” while performing the tasks; as the investigators believed that the tasks were not complex, they assumed that the verbalization of participants’ thought processes would not disrupt or interfere with their task performance. The performance measure was the time to complete or give up on the task. In addition, participants completed the 19-item Computer Usability Satisfaction Questionnaire (CUSQ; Lewis, 1995). Only two tasks were successfully completed by all participants: find password and login to the site, and locate and view a letter from the hospital. The tasks that produced significantly different completion times between rounds of testing were all related to locating and viewing information such as a radiology report. However, despite the modifications made between rounds that led to improvements in some of the task completion times, there were no significant differences in satisfaction in each of the 19 CUSQ items between the three rounds. Satisfaction was greatest for interface pleasantness and likeability and lowest for error messages and clarity of information. Generally, participants were reported to have problems stemming from medical terminology, portal navigation, and information overload, which the investigators noted were not uncovered in previous heuristic usability testing, focus groups, and questionnaire feedback from parents who were prior users of this system. However, the investigators did note that some of the problems encountered may have resulted from a lack of computer skills rather than portal issues (participants’ computer skills were not measured).

The usability of HealthView, the PHR of the Duke University Health System, was evaluated by Seagall et al. (2011) in a study that included 23 participants (mean age of 53), about 61% of whom had previously opened an account to this PHR. These participants, who were chronically ill patients and relatively inexperienced with computers, completed nine tasks in random order (e.g., finding and interpreting lab results and determining whether an allergy is documented in the record) and asked to “think aloud” as they carried out these tasks. Afterwards, participants were interviewed concerning usability problems they may have encountered. They also completed a usability survey related to their feelings about this PHR. Although ratings of usability were relatively high—eliciting, for example, strong agreement (in 96% of the participants) with statements such as “I can find information I need easily and quickly in HealthView”—these results were in contrast with the think aloud data, which revealed that navigation was problematic and that between 30%–60% of the participants experienced difficulty finding various types of information. Task performance measures consisted of the percentage of subtasks within each of the nine tasks that were given up on or in which an error was made (distinctions between these two metrics were not provided) and the percentage of the subtasks prompting help requests. Tasks with the highest rates of error or giving up involved making appointments (31.9%), finding and printing a payment history (25%), and determining how one’s weight has changed in the last year (22.4%). Although specific task completion time data were not reported, the investigators indicated that these times were generally long. The interview data included comments that this PHR was not a “walk-up-and-use” website and that the user interface “needs a lot of work.” Participants also raised issues regarding the presence within the PHR of unclear medical terminology. Recommendations provided by the investigators for improving the usability of this PHR focused on navigation, text salience, and added functionalities (such as electronic communication with patients’ providers).

In the study by Taha, Sharit, and Czaja (2014), 51 educationally and ethnically diverse older adults (mean age of 69) used a simulated PHR based on EPIC’s MyChart to perform 15 health management tasks commonly carried out using a patient portal (e.g., locating the date and time of an upcoming appointment and reviewing test results). These tasks were designed to span the spectrum of numeracy ability—basic, computational, analytical, and statistical numeracy—proposed by Golbeck, Ahlers-Schmidt, Paschal, & Dismuke (2005). Tests administered prior to task performance included the Test of Functional Health Literacy in Adults (TOFHLA; Parker, Baker, Williams, & Nurss, 1995) and a test of numeracy ability developed by Lipkus, Samsa, & Rimer (2001). Internet experience was also assessed. Following task performance, participants completed a 17-item usability questionnaire specifically designed for this study and were then interviewed. Tasks were categorized as either “simple” (seven tasks) or “complex” (eight tasks). The findings from the TOFHLA indicated very little variability among the participants in health literacy—about 85% of the participants scored in the “Adequate” range. However, the sample had objective numeracy scores that were quite low; about 53% of the participants could not correctly answer the majority of the numeracy questions. Mean task performance scores were 8.7 (out of a maximum of 14) on the simple tasks and 5.9 (out of a maximum of 16) on the complex tasks. Two regression models, one to predict simple task performance and one to predict complex task performance, were constructed. Education was not found to be a significant predictor in either model; the addition of Internet experience resulted in both models being significant, and objective numeracy was a significant predictor of both simple and complex task performance even after education and Internet experience were accounted for. Results from the usability questionnaire (Taha, 2012) indicated that the vast majority (94%) of participants thought that a PHR would improve their ability to perform health management tasks and 95% indicated that it would enable them to obtain information that would help them understand issues related to their health. However, participants also expressed some difficulty in using the PHR: 51% felt that it was difficult to locate the information that they needed and 40% indicated that they felt lost while navigating within the PHR. Participants also reported difficulty in comprehending information contained in the PHR, particularly information presented in numerical tables and in graphs. Recommendations for improvement provided by participants during the interviews included adding a drug interaction feature to the drug medications page and a line to each medication to remind the patient of its purpose; making links to view graphs of lab test results much more salient; and using less ambiguous or confusing terminology, for example, in discriminating between normal versus abnormal test results.

Health Literacy, Numeracy Ability, Graph Literacy and Use of PHRs

Although conventional interface design issues such as font size, organization of material, and navigation can affect the usability of any interface, and are especially relevant to older users (Fisk, Rogers, Charness, Czaja, & Sharit, 2009), as noted by Tang et al. (2006) individual factors such as health literacy, numeracy ability, and graph literacy are likely to play critical roles in the usability of PHRs. The basis for this assertion is that the interfaces associated with many PHRs, especially those that are tethered to electronic medical record systems, may require users to have special skills to use them given the complexity of the information these PHRs provide and the nature of the functionalities that they make available to users for accessing and manipulating this information. Measuring these individual factors within the context of performance-based usability studies thus provides the possibility for gaining a richer understanding of how the design of such PHRs can be improved for its users.

Health literacy has been defined as the “the capacity of individuals to obtain, interpret and understand basic health information and services and the competence to use such information and services in ways which are health enhancing” (Joint Committee on National Health Education Standards, 1995, p. 5). Patients with inadequate health literacy have been found to be less likely than patients with adequate health literacy to view laboratory results, send e-mails to providers, and make medical appointments using a patient portal tethered to their electronic health record (Sarkar et al., 2011). Two skills related to health literacy are numeracy and graph literacy. Health numeracy refers to the ability to understand and apply information that is conveyed with numbers, tables and graphs, probabilities, and statistics to effectively communicate with health care providers, take care of one’s health, and participate in medical decisions (Reyna, Nelson, Han, & Dieckmann, 2009). Graph literacy, which denotes the ability to understand basic graphical representations used to present quantitative information, has more recently emerged as another important skill needed for understanding and utilizing health care information (Galesic, Garcia-Retamero, & Gigerenzer, 2009; Garcia-Retamero, & Galesic, 2010). Given that much of the information communicated to a patient through a PHR is numeric and often provided in tabular and graphical formats, adequate numeracy and graph literacy skills would appear to be needed for successful engagement with PHRs. Furthermore, research has demonstrated a high prevalence of low health literacy (Martin et al., 2009; Morrow et al., 2006) and numeracy (Garcia-Retamero, Galesic, & Gigerenzer, 2010; Taha, Sharit, & Czaja, 2014) in older individuals, as well as high levels of inadequate health literacy, numeracy ability, and graph literacy among a sample of veterans in primary care (Rodríguez et al., 2013).

My HealtheVet

Veterans in the United States represent a particularly vulnerable population. Many of these individuals, especially older veterans (≥ 65 years of age), have multiple chronic conditions such as diabetes and hypertension and poor overall health status as reflected in both physical and emotional markers of health, and often maintain behavioral lifestyles that contribute to these health states (Selim et al., 2004). With the intention of improving the health and overall quality of life of veterans, the Department of Veterans Affairs (VA) introduced My HealtheVet (MHV) in 2003 as a web-based PHR to “complement traditional services, improve management of care, and empower patients and their families to play a more active role in veterans’ healthcare” (Nazi, 2010, p. 204). This PHR can be accessed from the Internet (www.myhealth.va.gov) by veterans, their families, and caregivers. Users can self-register to create a basic account, and VA patients can complete a one-time authentication process to upgrade to a Premium account that enables access to all MHV’s available features.

There are eight major categories or functionalities of MHV. These are designated in its main menu bar as Home, Personal Information, Pharmacy, Research Health, Get Care, Track Health, MHV Community, and Secure Messaging (Figure 1). Figures 2 and 3 illustrate the Vitals + Readings section of the 2012 version of the Track Health Function (THF), a particularly powerful tool for self-management of health. Using this tool, the patient can monitor and document various measures (e.g., blood pressure and blood sugar) with options to edit, delete, or add new and detailed information and to display information in tabular or graphical formats. In the Journals section of the THF, users can record their food intake and daily activity through corresponding activity and food journals (Figures 4 and 5).

Such human-computer interactive functionalities, however, are not likely to be adopted by MHV users unless, in addition to providing perceptible value to these users, these tools are also easy for them to use (Fisk et al., 2009). As stated by Tang et al. (2006), “In order to be useful to the patient, the PHR must present data and accompanying tools in ways that enable the individual to understand and to act on the information contained in the record. This is challenging because of patients’ widely varying levels of general literacy and health literacy” (p. 122). Findings from a relatively recent national survey study of veterans who are users of MHV hinted at this concern. The results from this survey indicated that most users visit the site primarily to use its pharmacy-related features (which enable, for example, prescriptions to be refilled) and that very little is known about why other functionalities are exploited less frequently (Nazi, 2010).

Figure 1. Home (log-in) page of My HealtheVet (MHV).

Figure 2. The Vitals + Readings section of the Track Health Function of MHV.

Figure 3. A graph view of a Track Health Function measure; the user could also see information in tabular formats.

Figure 4. Adding food information within the Food Journal of the Track Health Function.

Figure 5. Adding activity information within the Activity Journal of the Track Health Function.

Study Objectives

In this paper we report on a usability study directed at the 2012 version of MHV that comprised a sample of 40 veterans who were all authenticated users of MHV. The only known prior usability study of MHV targeted this portal’s less advanced 2008 version (Haggstrom et al., 2011). In that study, the performance of 28 veterans, all non-users of MHV, was assessed on three very basic tasks (including the ability to log into the website) that were required to be performed under highly specified time constraints. The present study was undertaken with the objective of determining how actual users of this PHR would fare in performing a much broader and more representative array of tasks that MHV currently offers its users, especially in light of recent findings (Nazi, 2010) that many of MHV’s functionalities, including those designed specifically to enhance self-management of health, are used relatively infrequently. The primary objectives of this study were (a) to examine the capabilities of these veterans to perform 13 health management tasks spanning all eight of MHV’s primary functions; (b) to determine the relative impacts of health literacy, numeracy ability, and graph literacy on task performance; (c) to identify variables that distinguish better performers from poorer performers; and (d) to determine if older veterans (≥ 65 years of age) perform more poorly than younger veterans and differ from them in ratings of the perceived usefulness and usability of this PHR.

Method

The following sections discuss the participants, measures, setting, and procedures used in this study. Details concerning task performance measures are discussed in a separate section.

Participants

Forty veterans, all receiving care at a VA medical center in Florida and who had already completed the one-time authentication process for use of MHV, were recruited in two age groups: < 65 years of age (n = 20, M = 51.85, SD = 6.48, and ranging from 37 to 63 years of age) and ≥ 65 years of age (n = 20, M = 71.0, SD = 7.12, and ranging from 65 to 92 years of age). The gender, race, and ethnicity distributions were as follows: 37 males and 3 females; 9 African Americans and 31 Whites; and 9 Hispanics and 31 non-Hispanics. The participants met the following inclusion criteria: enrollment in a VA clinic, cognitively intact (Mini-Cog > 3; Borson, Scanlan, Chen, & Ganguli, 2003), non-depressed (PHQ-2 < 3; Kroenke, Spitzer, & Williams, 2001), and having a minimum education level of eighth grade. The Mini-Cog is a well-accepted instrument used to screen for cognitive impairment in older adults in health care settings that takes about three minutes to administer. Compared to other similar screening tools, it is faster and considered less affected by a person’s ethnicity, language, and education. The PHQ-2 is a simple two-item instrument that screens for depression by inquiring about the degree to which an individual has experienced depressed mood over the past two weeks. Because the participants in this study were veterans, including older veterans, who were receiving health care at a VA facility, it was critical to screen for both cognitive impairment and depression as these conditions could adversely affect their performance on the health management tasks and thus undermine the assessment of the usability of MHV. The study received full IRB approval from the VA medical center, and all patients consented to participation through a consent form. Participants received a $10 voucher for participating in the study.

Measures

With the exception of the measures of median household income and comorbidity (the presence within an individual of simultaneous but independent adverse medical conditions, as described below), the following data were collected using paper-and-pencil questionnaire instruments. Collectively, these measures complement the measures of task performance and serve two primary purposes. First, they characterize the study sample on variables that could play a critical role in the ability to successfully use MHV. For example, very low scores overall on computer/Internet proficiency would suggest that these participants may be particularly challenged to use this portal. These measures also provide a basis for differentiating the younger from the older participants and thus for explaining age-group differences in task performance. Second, these measures enable theories to be tested regarding variables that may be contributing to lower as compared to higher task performance, consistent with our study objectives. This, in turn, could inform design interventions for improving the usability of this product.

Where applicable, values of Cronbach’s alpha (α) are reported for the measures. Cronbach’s α is a coefficient of internal consistency that generally increases as the intercorrelations among an instrument’s items increase, and it is commonly used as an estimate of the reliability of a test or questionnaire. Its theoretical value varies from zero to 1, with rules of thumb specifying values between 0.6 and 0.7 as acceptable, values between 0.7 and 0.9 as good, and values ≥ 0.9 as excellent (Kline, 2000).

Demographic information

Participants reported their age, level of education (secondary, technical, undergraduate, and graduate degrees), and ethnicity. To infer patients’ average household income we used the 5‑digit ZIP Code Tabulation Area (ZCTA) and the median household income in the past 12 months (in 2011 inflation-adjusted dollars) from the U.S. Census Bureau, 2007-2011, American Community Survey.

Charlson Comorbidity Index

Many veterans, especially older veterans, generally experience multiple adverse health conditions (i.e., comorbidities) such as heart disease, AIDS, or cancer that could negatively impact their ability to effectively use applications such as MHV. To account for the influence of this factor, the Charlson index (Charlson, Pompei, Ales, & MacKenzie, 1987), which is considered to be the most valid, reliable, and extensively studied comorbidity index for predicting the mortality of patients (de Groot, Beckerman, Lankhorst, & Bouter, 2003), was used to obtain comorbidity scores for each participant. This measure considers 22 health conditions. Each condition is assigned a score of 1, 2, 3, or 6, with higher scores reflecting higher risks of mortality associated with the condition. The scores are summed to provide a total score.

Computer/Internet status and proficiency

This instrument, which was developed for the purposes of this study, assessed (a) computer/Internet status through questions that asked participants if they had a computer with an Internet connection in their home and, if so, the extent to which they use it; and (b) computer/Internet proficiency through four questions that asked participants to rate their ability, on a 5-point scale ranging from none to excellent, to use features such as the mouse, keyboard, and email, and to use the Internet for searching information. The Cronbach α coefficient for the computer/Internet proficiency items was .91.

Health literacy

Health literacy was measured with the Newest Vital Sign (NVS), which consists of six questions about how one would interpret and act on the information contained on a nutrition label from an ice cream container (Weiss et al., 2005). Higher scores are indicative of greater health literacy. The NVS has demonstrated internal consistency in previous research (de Groot et al., 2003; Cronbach α of .76). In this study, the Cronbach α coefficient was .74.

Objective numeracy

Numeracy ability was measured with a scale consisting of nine items developed by Schwartz, Woloshin, Black, and Welch (1997) and Lipkus, Samsa, and Rimer (2001) and the four items of the Berlin Numeracy Test (Cokely, Galesic, Schulz, Ghazil, & Garcia-Retamero, 2012). The scale assesses the ability to compare risk magnitude, convert percentages to proportions, convert proportions to percentages, convert probabilities to proportions, and compute probabilities. The scale showed adequate internal consistency in previous research, with Cronbach’s α scores ranging from .70 to .75 (Cokely et al., 2012). The Cronbach α coefficient was .80 in this study.

Graph literacy

To measure graph literacy, the graph literacy scale developed by Galesic and Garcia-Retamero (2011) was used. It consists of 13 items and measures three abilities: finding specific information in the graph, finding relationships in the data as shown on the graph, and making inferences and predictions from the data. The graph literacy scale has been subjected to validation on probabilistic national samples in Germany and the United States (Galesic & Garcia-Retamero, 2011). The Cronbach α coefficient was .63 in this study.

Perceived usefulness and usability of MHV

This instrument, which was developed for the purposes of this study, assessed five items pertaining to MHV usefulness (e.g., “I like all the different health management tasks you can do with My HealtheVet”) and seven items pertaining to MHV usability (e.g., “I find it easy to get lost when navigating around My HealtheVet”) on a 5-point scale ranging from strongly agree to strongly disagree, resulting in item scores ranging from 0–4. The content validity of the items and their categorization as markers of either usefulness or usability of MHV were established through agreement by three of the study investigators. The Cronbach α coefficients were .70 for the perceived usefulness items and .86 for the perceived usability items.

We chose not to use a short, easily available, and already validated scale such as the System Usability Scale (SUS; Brooke, 1986) for several reasons. First, although the SUS addresses usability, we wanted an instrument that captures perceptions regarding the usefulness of this portal as well as its usability as the relative lack of use of MHV by some of these users might be due to perceptions that this product does not provide sufficient benefits to warrant greater utilization. Second, some of the questions in the SUS use terminology or phrases that may be misunderstood by some of the participants from this population of users (e.g., “…various functions were well integrated,” and “I would imagine that most people”). Finally, we wanted to address specific issues that we uncovered in our own heuristic evaluation of MHV. Although we recognize that a benefit of the SUS is that it allows comparisons to be made with other products, whether similar or dissimilar, this characteristic is also a drawback as the ability to make such comparisons requires the SUS to be very general in nature. Thus, we elected to customize our questionnaire to MHV and its user population.

Self-reported use and confidence in use of MHV

This questionnaire, which was developed for the purposes of this study, measured use of MHV by a question that asked participants to indicate the extent to which they use each of the eight MHV menu categories on a 5-point scale ranging from never to a lot. Confidence in use of MHV was measured by a second question that asked participants to indicate how confident they are in their ability to use each of the eight MHV categories on a 5-point scale ranging from not at all confident to very confident. This instrument yielded Cronbach α coefficients of .82 for self-reported use of MHV and .95 for self-reported confidence in use of MHV.

Setting

The study was conducted in a single quiet room within the Laboratory of E-learning and Multimedia Research at the VA medical center in Florida. Participation occurred on an individual basis, with each participant seated in front of an HP Touchsmart 1300 PC with a 23-inch LCD screen. The research coordinator was present in the same room for the duration of the session, in case of queries or other issues.

Procedure

Participants first completed all the questionnaires. Following a rest break, they were asked to perform a set of 13 MHV health management tasks (see Table 1). Task performance was limited to two hours, which provided sufficient time for all the participants to attempt the set of tasks. MHV accounts were pre-populated by fictitious patient data as determined by two of the research team’s physicians. The task statements were presented in sequence, on paper, but were not required to be performed in their listed order. Prior to performing the tasks participants were told that eye-tracking data would be collected from them (these data are not reported here); the eye-tracking system, however, did not require the participant to wear any type of head covering or other markers or to be restrained. After completion of the tasks, exit interviews were administered to each participant. These interviews were audio-recorded and transcribed and coded using the qualitative data analysis software application Atlas.ti (http://www.atlasti.com/index.html).

Task Performance Measures

Performance of each of the 13 tasks was evaluated either based on whether the correct answer to the question was obtained (for example, in the case of task 3 in Table 1) or whether the actions instructed by the task were correctly performed (for example, in the case of task 4 in Table 1). The data used for performance evaluation in the first situation derived from information the participant was instructed to write down on paper; the data used for performance evaluation in the second situation derived from the eye-tracking system’s video features.

Based on these data, each task was scored as incorrect, partially correct, or correct. A task was scored as incorrect if the participant indicated that he or she was unable to perform the task, provided incorrect information on the paper, or performed actions that were incorrect. A score of zero was assigned to any task evaluated as incorrect. A task was scored as partially correct if part of the information provided or some of the actions performed (in the case of tasks evaluated based on actions undertaken) were correct. A score of one was assigned to any task evaluated as partially correct. A task was scored as correct if the information provided was correct or the actions performed (in the case of tasks evaluated based on actions undertaken) were correct. A score of two was assigned to any task evaluated as correct. Thus, each task was assigned a score of zero, one, or two. A total task performance score was computed for each participant; this score was defined as the sum of the individual task scores. As there were 13 tasks, and each individual task could be assigned either a score of zero, one, or two, the minimum total task performance score possible was zero and the maximum total task performance score was 26.

The primary measure of task performance was the total task performance score (with a range of 0–26). Two additional measures of task performance were also computed. The first of these was simply a count, for each participant, of the number of the 13 tasks that they completed correctly (i.e., had a score of two assigned). As participants were required to complete 13 tasks, the score for this measure could only range between zero and 13. The second additional measure of task performance was the MHV category performance score. This score, like the total task performance score, was also based on the scores assigned to the individual tasks. If the MHV category was only represented by one task, then the score for that individual task corresponded to the performance score for that MHV category as well. However, if the MHV category was represented by more than one task (for example, as shown in Table 1, there were four tasks associated with the Track Health category of MHV), then a participant’s MHV performance score for that category was defined as the mean of the participant’s individual task performance scores comprising that category.

Verbal data are often derived in usability studies based on the “think aloud” method, whereby participants provide their thoughts as they carry out tasks. However, due to the nature of this population of users, which included older veterans, and the fact that some of the tasks were reasonably challenging, we chose not to use this method given the belief that the requirement of thinking aloud while negotiating these tasks might impose excessive cognitive load and thus interfere with task performance.

Results

Chi-square tests indicated no significant age-group differences for this group of veterans in race, ethnicity, or education. Table 2 presents the means and standard deviations for the study measures. There were no significant age-group differences in self-reported use of MHV (nor were there any age-group differences for any of the individual MHV menu categories) or in self-reported confidence in use of MHV. Significant age-group differences (p < .05), however, were found for the Charlson Comorbidity Index, t(38) = 2.83, p = .022; health literacy, t(38) = 2.16, p = .038; total task performance score, t(38) = 3.26, p = .002; and the number of tasks completed correctly, t(38) = 3.14, p = .003. In each of these cases, the older age-group participants had poorer scores. Analysis of the MHV category performance scores computed for each of the eight MHV categories indicated that the older age-group participants performed significantly worse in four MHV categories: Get Care, t(38) = 2.26, p = .03; Research Health, t(38) = 2.03, p = .05; Track Health, t(38)= 3.14, p = .003; and Secure Messaging, t(38) = 2.79, p = .01.

A total task performance score of 17 provided an approximate median split that enabled 19 lower performers to be contrasted with 21 higher performers. As expected, the performance differences between these two groups were significant (Table 3): t(38) = 9.47, p < .001 for the total task performance score and t(38) = 9.25, p < .001 for the number of correct tasks. Although on average the better performers were younger, t(38) = 3.07, p = .004, five of the lower performers were < 65 years of age while six of the higher performers were ≥ 65. In addition, health literacy, t(38) = 3.72, p = .001; objective numeracy, t(38) = 2.93, p = .006; and graph literacy, t(38) = 3.28, p = .002, were all diagnostic in differentiating the lower from the higher task performers. The higher performers also perceived MHV as more usable than the lower performers, t(38) = 2.10, p = .043. This latter result indicates that at least for this sample of users there appears to be a positive relationship between perceptions of usability and performance, which is usually but not always found in usability studies (Sauro & Lewis, 2009).

Table 1. The 13 Tasks Used for MHV Usability Assessment

Table 2. Means and Standard Deviations of the Study Measures

p < .05; **p < .01; based on independent two-sided t-tests.

Table 3. Means and Standard Deviations of Measures for the Lower and Higher Task Performers^a

^aA total task performance score of 17 was used to achieve an approximate median split. *p < .05; **p < .01; ***p < .001; based on independent two-sided t-tests.

Hierarchical regression models were constructed to determine if health literacy, numeracy ability, and graph literacy had significant impacts on performance after controlling for computer/Internet proficiency and comorbidity scores. The five predictor variables were entered in the following order: computer/Internet proficiency, Charlson Comorbidity Index, health literacy, objective numeracy, and graph literacy. The results of this analysis (Table 4) indicated that even after controlling for computer/Internet proficiency and comorbidities, health literacy still accounted for a significant amount of the variability in task performance. In addition, graph literacy was a significant predictor even after the other four model predictors were accounted for. In fact, despite the finding of no significant age-group differences in graph literacy, this factor was the single most powerful model predictor, accounting by itself for 39.1% of the variance in the total task performance measure.

Table 4. Hierarchical Regression Models for the Total Task Performance Measure

1. computer/Internet proficiency
2. computer/Internet proficiency, Charlson Comorbidity Index
3. computer/Internet proficiency, Charlson Comorbidity Index, health literacy
4. computer/Internet proficiency, Charlson Comorbidity Index, health literacy, objective numeracy
5. computer/Internet proficiency, Charlson Comorbidity Index, health literacy, objective numeracy, graph literacy

 

Exit interviews

The exit interviews revealed a number of concerns that these participants, who were all authenticated users of MHV, had with this PHR. Participants also often offered a number of their own recommendations for design improvements. For example, a typical concern related to having to cope with too much information was expressed by one participant as follows:

For me, personally one of the biggest issues…is when you go there, there is always too much information…it is just harder to find what I want to find. Let us make it simple and straightforward. I think we need to design this for people that basically are not very computer literate, but then do not make it so ponderous that people who are [computer literate] say “I am not going to mess with this because it takes forever” and I do not know how you get that, you know that middle ground, but in my opinion drop down menus would simplify a lot of things.

This general concern with having to cope with too much information, but with an emphasis on navigational issues, was expressed as follows by another participant:

I find that some of the information is too excessive, like it is just too much information…as long as you categorize it, if the webpage is very simple with specific links that take you to specific general areas and then once you click that link it breaks down to more specific areas, then you are not going to really run the risk of exhausting the person. You exhaust the person when they have to go and link and link and link and going in circles and circles and I think it has gotten to a point where it really needs to be redesigned in a more user friendly way, in terms of [a] hierarchy of information.

Another participant conveyed the general issue of navigation very succinctly: “Just going from page one to page two I just was already lost.” In response to the interviewer posing the question “So navigating is hard, going back and forth was hard?” this participant responded “Yes navigating, that is the word.”

One participant offered the following suggestion for mitigating issues associated with finding information:

Perhaps an index or a map of where everything is…a more intuitive sense of where to find things. Although I have been on the website before I did not have a real concise knowledge of where everything was. So, maybe if they have tabs or something where you could look at a glance and see where to find things.

Another participant suggested the need for tutorials to help users based on the following concern:

There is no pre-knowledge [of] how to use this system, so you need like a section where you can come in and watch a video or something with [a] demonstration, showing [you] how to use the different things in the program.

When the interviewer responded with “like a tutorial,” the participant concurred “Right, like a tutorial, you need a tutorial.”

With respect to the issue of visibility, one participant indicated the following:

The words you can read but the toolbars are very small…when you are looking at the page there are much bigger things than that little toolbar, so the toolbar is the main map, it is the map where you are going, [and] it was kind of small.

However, other participants did express frustration regarding the relatively small size of the text in addition to the size of screen objects.

Discussion

Based on the performance of 13 tasks encompassing the eight major categories of MHV by veterans who were all users of MHV, the findings indicated that overall the younger participants performed reasonably well, especially considering that they reported relatively low degrees of usage on a number of the functionalities from which these tasks were drawn. In contrast, the older veterans performed significantly worse. Based on their average total task performance score of 12.7 and a maximum achievable score of 26, they attained a total task performance score that, relative to the maximum score, was only about 49%, despite not reporting having had significantly less prior experience than their younger counterparts on any of these eight MHV categories. Moreover, the MHV categories on which the older veterans performed significantly worse are potentially critical for self-management of health, implying that these users of MHV may be particularly compromised with respect to the likely benefits that this patient portal has to offer.

Similar to the participants in the usability studies reviewed earlier in this paper, the participants in this study also alluded (during their exit interviews) to problems they had with navigation within this portal and with finding information due to issues related to information organization and overload. MHV is clearly a complex application that imposed on its designers the challenge of providing large amounts of various types of information through a host of functionalities in order to serve the diverse health-related needs of both younger and older veterans. However, unlike many other health websites, PHRs (such as MHV) that are tethered to electronic medical record systems provide unique challenges for this population of users. For example, features such as its Track Health Function require that the user be able to input various kinds of data in both numeric and text formats; move rapidly between sections of this menu category in order to integrate information between data in food and activity journals, lab test results, and physiological measures such as weight and blood pressure; and generate information from tables and graphs over different timelines while also being able to visualize and comprehend this information. Other MHV functionalities, such as Research Health that provide links to documents that provide information tailored to veterans’ health issues as well as to websites such as WebMD so that users can, for instance, research potential side effects of a medication that they have been prescribed, have their own set of issues related to finding information and staying oriented—i.e., not losing track of where one is and knowing how to get back to some other place within the website.

Prior to this study, our own heuristic analysis of MHV in fact revealed the existence of potential problems associated with navigation between the various menu categories and sections within these categories that were indeed capable of inducing such loss of orientation. Some of these problems stemmed from inconsistencies found in the way sidebars were used to support movements across the various entities within the portal. Thus, improvements to the interface to MHV should include the creation of a clearer and more consistent process of navigation that would enable its users to easily and quickly move between sections of this PHR, especially across those sections that require the user to integrate information. Providing a type of “roadmap” within the application that could offer users a better intuitive grasp of where they are relative to where they need to go, as suggested by one of the participants; using text redundant with icons that symbolize the various locations; and using an efficient means for reaching destinations without being locked into a succession of links, could be helpful to many users.

Design interventions should also be directed at making user input of textual and numeric health-related information a process that is easier to perform. This can be accomplished, for example, through screen areas that become highlighted and enlarged in separate boxes when the user selects an area designated for data input, thus making data entry less visually taxing and error prone. Also critical is providing the necessary organization and salience of information on the website pages so that needed information is not only easier to find but is also visually appealing, as a good first impression when encountering information within health websites has been found to be a critical factor in engendering its use (Sillence, Briggs, Harris, & Fishwick, 2007). It is important, though, that the implementation of any interface design strategy is compatible with the normal age-related declines in cognitive processing of information (Czaja & Sharit, 2012) that many older veterans are apt to be experiencing. The benefits of such redesign efforts, however, are likely to be pervasive as a design optimized for an older adult population generally leads to improved usability for the larger population of users (Fisk et al., 2009).

The findings from this study also point to additional considerations that could impact not only the usability of this portal, but also other similar applications, including health-related websites that require their users to comprehend numeric health information in textual, tabular, or graphical formats (e.g., Sharit, Hernandez, Nair, Kuhn, & Czaja, 2011). One finding, which was consistent with the literature, was the significantly lower health literacy among the older participants, which may have contributed to adversely impacting their ability to make effective use of this PHR. Also, the profiles of the lower and higher performers (Table 3) revealed that in addition to one’s age, health literacy, numeracy ability, and graph literacy were also influential individual factors in differentiating these two groups of performers. Finally, hierarchical regression analysis highlighted the unique contribution of graph literacy that, when considered alone, was the most diagnostic predictor of performance. Notably, of the eight primary MHV categories, graph literacy correlated most strongly with performance on the Track Health Function (r = .61, p < .01), a functionality that can place substantial visual-spatial demands on users.

Given the potential impact that these individual variables can have in making effective use of these types of PHRs, improvements to the interface should thus also consider making embedded aids available to help users overcome lower skills in health literacy, health numeracy, and graph literacy. For example, one such aid can be in the form of an automated glossary that provides simple translations of medical terminology in a pop-up box when the user places the mouse cursor over a word. Other aids can offer analogies (Galesic & Garcia-Retamero, 2013) to help users comprehend various numeric concepts, such as the concept of a trend (for example, to answer questions related to whether one’s cholesterol or blood sugar level has been improving over the past six months) or the concept of a cause-effect relationship (for example, to answer questions related to whether changes in one’s diet or physical activity are related to improvements in physiological indices). These aids should be easily accessible if desired and not intrusive so that they do not induce distraction, and can be presented in multimedia formats using voice coupled with visual textual and pictorial information.

With respect to strengths and limitations of this study, some of the strengths include the use of authenticated users of MHV; a comprehensive set of health-management tasks for testing the performance capabilities of MHV users; an array of validated scales to enable the contribution of factors such as health literacy, objective numeracy, and graph literacy to be assessed; and the supplementary use of chart audit data to obtain data on the status of the participants’ health conditions. Limitations of this study include the use of a convenience sample of primary care mostly male veterans recruited at one VA medical facility.

Key Points

The following are the main points raised in this study:

• Patient portals such as the VA’s My HealtheVet (MHV) can potentially enable veterans to greatly improve the management of their own health, but its interface to the array of functionalities it offers may require that its users have health literacy, numeracy, and graph literacy skills.
• Despite reporting similar degrees of use of the various functionalities of MHV, older veterans performed significantly worse than veterans who were on average about two decades younger on a set of health management tasks, and particularly on functionalities that are critical for tracking and managing one’s health.
• The differences in task performance scores between the 19 lower performers and the 21 higher performers were relatively extreme, and health literacy, numeracy ability, graph literacy, and perceptions of usability all proved significant in differentiating these two groups of performers. Graph literacy was a significant predictor of performance even after accounting for computer/Internet proficiency, comorbidity scores, health literacy, and numeracy ability.
• Modifying the interface of MHV or other similar applications in order to reduce their dependency for successful use on factors such as numeracy ability and graph literacy, in addition to making information easier to find, visualize, and navigate, is challenging, but a number of strategies offer promise and should be considered for evaluation.

Conclusions

In conclusion, the findings from this research imply that veterans with low computer proficiency skills, older veterans, and MHV users with low health literacy, numeracy, and graph literacy skills may be at a disadvantage in attaining the array of health management benefits that MHV has to offer. The translation of these findings into interface design strategies, while challenging, are timely in light of the highly anticipated MHV redesign efforts planned by the VA. The goal is to use these insights to directly inform the VA’s efforts to continue to improve the usability of MHV for all veterans, which can also translate to improved use of PHRs by the general population.

References

• Administration on Aging, U.S. Department of Health and Human Services. (2011). A profile of older Americans: 2011. Retrieved from http://www.aoa.gov/AoARoot/Aging_Statistics/Profile/index.aspx
• Archer, N., Fevrier-Thomas, V., Lokker, C., McKibbon, K. A., & Straus, S. E. (2011). Personal health records: A scoping review. Journal of the American Medical Informatics Association, 18, 515-522.
• Borson, S., Scanlan, J. M., Chen, P., & Ganguli, M. (2003). The Mini-Cog as a screen for dementia: Validation in a population-based sample. Journal of the American Geriatrics Society, 51, 1451-1454.
• Britto, M. T., Jimison, H. B., Munafo, J. K., Wissman, J., Rogers, M. L., & Hersh, W. (2009). Usability testing finds problems for novice users of pediatric portals. Journal of the American Medical Informatics Association, 16(5), 660-669.
• Brooke, J. (1996). SUS: A “quick” and “dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor & Francis.
• Charlson, M. E., Pompei, P., Ales, K. L., & MacKenzie, C. R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40(5), 373-383.
• Cokely, E. T., Galesic, M., Schulz, E., Ghazil, S., & Garcia-Retamero, R. (2012). Measuring risk literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25-47.
• Czaja, S. J., & Sharit, J. (2012). Designing training and instructional programs for older adults. Boca Raton, FL: CRC Press.
• de Groot, V., Beckerman, H., Lankhorst, G. J., & Bouter, L. M. (2003). How to measure comorbidity: A critical review of available methods. Journal of Clinical Epidemiology, 56(3), 221-229.
• Fisk, A. D., Rogers, W. A., Charness, N., Czaja, S. J., & Sharit, J. (2009). Designing for older adults: Principles & creative human factors approaches (2nd ed.). Boca Raton, FL: CRC Press.
• Galesic, M., & Garcia-Retamero, R. (2011). Graph literacy: A cross-cultural comparison. Medical Decision Making, 31(3), 444–457. doi: 10.1177/0272989X10373805
• Galesic, M., & Garcia-Retamero, R. (2013). Using analogies to communicate information about health risks. Applied Cognitive Psychology, 27(1), 33-42. doi: 10.1002/acp.2866 36
• Galesic, M., Garcia-Retamero, R., & Gigerenzer, G. (2009). Using icon arrays to communicate medical risks: overcoming low numeracy. Health Psychology, 28(2), 210-216. doi:10.1037/a0014474
• Garcia-Retamero, R., & Galesic, M. (2010). Who profits from visual aids? Overcoming challenges in people’s understanding of risks. Social Science & Medicine, 70(7), 1019-1025. doi:10.1016/j.socscimed.2010.03.019
• Garcia-Retamero, R., Galesic, M., & Gigerenzer, G. (2010). Do icon arrays help reduce denominator neglect? Medical Decision Making, 30(6), 672-684. doi:10.1177/0272989X10369000
• Golbeck, A. L., Ahlers-Schmidt, C. R., Paschal, A. M., Dismuke, S. E. (2005). A definition and operational framework for health numeracy. American Journal of Preventive Medicine, 29 (4), 375-376. doi:10.1016/j.amepre.2005.06.012
• Haggstrom, D. A., Saleem, J. J., Russ, A. L., Jones, J., Russell, S. A., & Chumbler, N. R. (2011). Lessons learned from usability testing of the VA’s personal health record. Journal of the American Medical Informatics Association, 18(Supplement 1):i13-17. doi:10.1136/amiajnl-2010-000082
• Joint Committee on National Health Education Standards (1995). National health education standards: Achieving health literacy, Atlanta, GA: American Cancer Society.
• Kline, P. (2000). The handbook of psychological testing (2nd ed.) London: Routledge.
• Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The Patient Health Questionnaire-2: Validity of a two-item depression screener. Medical Care, 41(11), 1284-1292.
• Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human Computer Interaction, 7(1), 57-78.
• Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21(1), 37-44. doi: 10.1177/0272989X0102100105
• Markle Foundation. (2011). PHR adoption on the rise. Retrieved from http://www.markle.org/publications/1440-phr-adoption-rise
• Martin, L. T., Ruder, T., Escarce, J. J., Ghosh-Dastidar, B., Sherman, D., Elliot, M., Bird, C. E., Fremont, A., Gasper, C., Culbert, A., & Lurie, N. (2009). Developing predictive models of health literacy. Journal of General Internal Medicine, 24(11), 1211-1216. doi: 10.1007/s11606-009-1105-7
• Morrow, D., Clark, D., Tu, W., Wu, J., Weiner, M., Steinley, D., & Murray, M. D. (2006). Correlates of health literacy in patients with chronic heart failure. Gerontologist, 46(5), 669-676.
• Nazi, K. M. (2010). Veterans’ voices: Use of the American Customer Satisfaction Index (ACSI) Survey to identify MyHealtheVet personal health record users’ characteristics, needs, and preferences. Journal of the American Medical Informatics Association, 17, 203-211.
• Parker, R. M., Baker, D. W., Williams, M. V., & Nurss, J. R. (1995). The test of functional health literacy in adults: A new instrument measuring patients’ literacy skills. Journal of General Internal Medicine, 10(10), 537-541. doi:10.1007/BF02640361.
• Ralston, J. D., Hirsch, I. B., Hoath, J., Mullen, M., Cheadle, A., & Goldberg, H. I. (2009). Web-based collaborative care for type 2 diabetes: A pilot randomized trial. Diabetes Care, 32(2), 234-239. doi: 10.2337/dc08-1220
• Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F. (2009). How numeracy influences risk comprehension and medical decision making. Psychological Bulletin 135(6), 943-973 doi: 10.1037/a0017327.
• Rodríguez, V., Andrade, A. D., García-Retamero, R., Ramanakumar, A., Rodríguez, R., Lisigurski, M., Sharit, J., & Ruiz, J. G. (2013). Health literacy, numeracy, and graphical literacy among veterans in primary care and their effect on shared decision-making and trust in physicians. Journal of Health Communication: International Perspectives, 18, Supplement 1, 273-289. doi:10.1080/10810730.2013.829137
• Sarkar, U., Karter, A. J., Liu, J. Y., Adler, N. E., Nquyen, R., López, A., & Schillinger, D. (2011). Social disparities in Internet patient portal use in diabetes: Evidence that the digital divide extends beyond access. Journal of the American Medical Informatics Association, 18, 318-321 doi: 10.1136/jamia.2010.006015
• Sauro, J., & Lewis, J. R. (2009). Correlations among prototypical usability metrics: Evidence for the construct of usability. CHI 2009, April 4-9, Boston, MA (pp. 1609-1618). New York, NY: ACM.
• Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine, 127(11), 966-972.
• Segall, N., Saville, J. G., L’Engle, P., Carlson, B., Wright, M. C., Schulman, K., & Tcheng, J. E. (2011). Usability evaluation of a personal health record. Proceedings of the AMIA 35th Annual Symposium on Biomedical and Health Informatics (pp. 1233-1242).
• Selim, A. J., Berlowitz, D. R., Fincke, G., Cong, Z., Rogers, W., Haffer, S. C., Ren, X. S., Lee, A., Qian, S. X., Miller, D. R., Spiro, A., Selim, B. J. & Kazis, L. E. (2004). The health status of elderly veteran enrollees in the Veterans Health Administration. Journal of the American Geriatrics Society, 52(8), 1271-1276. doi: 10.1111/j.1532-5415.2004.52355.x
• Sharit, J., Hernandez, M. A., Nair, S. N., Kuhn, T., & Czaja, S. J. (2011). Health problem solving by older persons using a complex government website: Analysis and implications for web design. ACM Transactions on Accessible Computing, 3, Article 11, 1-35.
• Sillence, E., Briggs, P., Harris, P., & Fishwick, L. (2007). Health websites that people can trust—The case of hypertension. Interacting with Computers, 19, 32-42.
• Taha, J. (2012). The effects of age, cognition, and health numeracy on use of a patient portal of an electronic medical record (EMR). Open Access Dissertations, Paper 814.
• Taha, J., Sharit, J., & Czaja, S. J. (2014). The impact of numeracy ability and technology skills on older adults’ performance of health management tasks using a patient portal. Journal of Applied Gerontology. 33(4), 415-435. doi: 10.1177/0733464812447283
• Tang, P. C., Ash, J. S., Bates, D. W., Overhage, M., & Sands, D. Z. (2006). Personal health records: Definitions, benefits, and strategies for overcoming barriers to adoption. Journal of the American Medical Informatics Association, 13(2), 121-126. doi: 10.1197/jamia.M2025
• Teneforde, M., Jain, A., & Hickner, J. (2011). The value of personal health records for chronic disease management: What do we know? Family Medicine, 43(5). 351-354.
• Weiss, B. D., Mays, M. Z., Martz, W., Castro, K. M., DeWalt, D. A., Pignone, M. P., Mockbee, J., & Hale, F. A. (2005). Quick assessment of literacy in primary care: The newest vital sign. Annals of Family Medicine, 3(6), 514-522.
• Wynia, M., & Dunn, K. (2010). Dreams and nightmares: Practical and ethical issues for patients and physicians using personal health records. Journal of Law, Medicine & Ethics, 38(1), 64-73. doi: 10.1111/j.1748-720X.2010.00467.x
• Undem, T. (2010). Consumers and health information technology: A national survey. Berkley: California HealthCare Foundation.

Published in: in Volume 9, Issue 4,