If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Rehabilitation Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, AmsterdamDepartment of MS Expertise Center, Nieuw Unicum, Zandvoort
Department of Rehabilitation Medicine, MS Center Amsterdam, Amsterdam Public Health, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam
Department of Rehabilitation Medicine, MS Center Amsterdam, Amsterdam Public Health, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam
Department of Rehabilitation Medicine, MS Center Amsterdam, Amsterdam Public Health, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam
Department of Rehabilitation Medicine, MS Center Amsterdam, Amsterdam Movement Sciences, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
The Whereabouts app creates a unique individual societal participation profile.
•
The app creates a real-time timeline of specific participation activities.
•
The app assess participation with ratings of meaningfulness and perceived strain.
•
The app is a potential valuable tool for patients, clinicians, and researchers.
Abstract
Objective
To develop a mobile health app to assess individual-specific meaningful societal participation in real time and to evaluate its usability.
Design
Development and usability study.
Setting
General community.
Participants
Persons with multiple sclerosis (PwMS) utilized the app for 7 consecutive days. In total, 72 PwMS and smartphone owners were included in the analysis (N=72).
Interventions
Using location tracking, the newly developed Whereabouts app generates an individual-specific timeline of societal participation activities each day, consisting of location and transportation intervals. Subsequently, this timeline is real time enriched by the user with self-reported ratings of meaningfulness and perceived strain of these societal participation activities. The app is based on the International Classification of Functioning, Disability and Health and was developed in an iterative process.
Main Outcome Measure(s)
Usability of the newly developed Whereabouts app was evaluated by analyzing the effectiveness, efficiency, and user satisfaction.
Results
Regarding effectiveness, the app correctly assessed the type, frequency, and duration of different societal participation activities for 96.1% of the participation activities. The self-reported ratings of societal participation varied for meaningfulness (range, 5-8), perceived strain (range, 2-6), and meaningfulness of the perceived strain (range, 5-8). The latter 2 were highly correlated (r=0.857). With regards to efficiency, 3.9% of the generated participation activities had to be excluded due to inaccuracy or incompleteness. Relating to user satisfaction, 57.3% of PwMS reported that they were satisfied with the usability of the app and 59.7% stated that it gave a realistic overview of their daily activities. However, 54.4% PwMS missed the possibility to specify activities at home, to add activities in more detail, and to correct mistakes.
Conclusions
The Whereabouts app demonstrates usability in assessing real-time, individual-specific meaningful societal participation. Improvements are recommended, such as the possibility to specify participation activities and to generate a graphic overview.
Multiple physical, cultural, and social barriers limit participation of people with disabilities, including those with multiple sclerosis (MS). The effect of MS-related symptoms on societal participation can be different in time and context and is unique for each person due to different perceptions of what aspects of societal participation are perceived as important.
According to the International Classification of Functioning, Disability and Health (ICF), participation is defined as “involvement in a life situation,” describing what an individual does in their current environment that brings in a societal context.
(p127) The assessment of societal participation is complicated by limitations such as recall bias, large intra- and interindividual variance in personal context, and, consequently, relative lack of responsiveness toward treatment effects. Variations in and between personal preferences and values over time of individual societal participation, like satisfaction, meaningfulness, or relevance of a particular activity, are still difficult to measure.
Mobile health (mHealth) apps can be useful in assessing societal participation and have become increasingly popular in recent years to capture interindividual fluctuations in physiological or psychological parameters.
Currently, the majority of mHealth apps for MS consists of patient education features. Remote monitoring and energy and resource management are often not present, and involvement of health care professionals is lacking.
or using a smartwatch with Global Positioning System (GPS) to assess walking in daily life, but no association has been made with societal participation.
The aim of the present study was to develop a mHealth app to assess individual-specific meaningful societal participation in real time and to evaluate its usability.
Methods
Overview of the Whereabouts app
Information about the developmental process of the Whereabouts-app can be found in the supplemental appendix S1 (available online only at http://www.archives-pmr.org/). Figure 1 provides an overview of the Whereabouts app (version 1.0.0). The Whereabouts app is based on participation domains of the ICF (see table 1).
Transport domains were added to provide insight into the way of transportation between societal participation on location.
Fig 1Overview of the Whereabouts application, translated from Dutch. (a) Timeline of 1 day filled in by a user, (b) question about the societal participation domains, and (c-e) question about the value of the domain.
Table 1Societal participation domains of the app in relation to the ICF activities and participation categories
Whereabouts Domains
ICF Domains
Carrying out daily routines (planning and organizing of daily living and manage expenses)
D2. General tasks and demands
Transport: walking, bicycle/hand bike, mobility scooter, wheelchair, car as driver, car as passenger/taxi/supplementary public transport, public transport (train/bus/tram), other
D4. Mobility
Personal care
D5. Self-care
Health care utilization
D5. Self-care
Household tasks/groceries/gardening
D6. Domestic life
Caring for others
D6. Domestic life
Social interaction and relationships
D7. Interpersonal interactions and relationships
Work/education
D8. Major areas (eg, work)
Recreation/leisure/sport
D9. Community, social, and civic life
Whereabouts questions
Question 1: What did you mainly do at this location?
Question 2: How meaningful is this activity to you?
Question 3: How much strain does this activity take?
Software is usable if it allows the user to execute their task effectively, efficiently, and with satisfaction in the specified context of use (ISO-9241).
Following this standard, 3 attributes of usability were assessed: (1) effectiveness; that is, how well do the users achieve their goals using the system; (2) efficiency; that is, what resources are consumed in order to achieve their goals; and (3) satisfaction; that is, how do the users feel about their use of the system.
in: Turner T. Szwillus G. Czerwinski M. Paternò F. Pemberton S. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems–CHI ’00. The Hague. Association for Computing Machinery,
The Netherlands2000: 345-352
To assess whether the content of the app contains real-time and individual-specific constructs, we tested whether the Whereabouts app was able to adequately assess the participation activities and the individual's appreciation of the engagement in these activities. We analyzed the types and frequency of segments with a differentiation in at home, transport, and on location. We evaluated overall duration in hours of societal participation activities on location and transport.
Efficiency
As indicators of efficiency, the accuracy and completeness of the segments were used. Quantification of the accuracy of the constantly moving transport segments was done by using the precision radius of GPS. This is defined as a 68% probability that the true location is inside that radius.
Exclusion of segments was done when the precision radius was ≥80 m, if segments were noted as inaccurate, or if they were not filled in by users (considered as inadequate or missing). Outlying segments (ie, lasting more than 8 hours) were manually verified by 2 researchers. By analyzing the completeness of generated segments with less than 25% of battery, the balance between smartphone battery and app performance was evaluated. Users’ feedback made in notes about errors and positive and negative characteristics of the app was analyzed.
User satisfaction
A self-developed evaluation form with rating scales and open questions was used as an indicator of user satisfaction.
Procedure
The study population, recruited between March 2019 and August 2019 in The Netherlands via the MS Center Amsterdam and via local MS organizations, consisted of PwMS with definite MS, aged ≥18, who were smartphone owners (Android 5+ or IOS 9+). Thirteen potential participants did not fulfill these criteria and were excluded from participation (see figure 2).
This is an explorative study in which the development and usability of a newly developed mobile application is investigated. A metastudy on the effectiveness of usability evaluation suggested 10±2 as a general rule for optimal sample size.
In a next step, validation of the app will be explored. Therefore, a sample size of 80 PwMS should allow for sufficient statistical power to assess the association between societal participation and mobility or make between-group comparisons of fatigued vs nonfatigued participants.
Ethical procedure
The Medical Ethical Committee of Amsterdam UMC, location VUmc, declared that the Medical Research Involving Human Subjects Act does not apply to this study and this study complies with the General Data Protection Regulation (reference number 2018.677).
An informed consent form and a digital questionnaire to assess demographic, disease, and disability characteristics were completed. During a home visit, the Whereabouts app was installed and an instruction leaflet was given. Cognition was measured with the Processing Speed Test.
The Processing Speed Test is an iPad-based screening instrument that measures the level of processing speed as an indicator for cognitive functioning. Participants used the Whereabouts app during a monitoring period of 7 consecutive days.
Statistical analyses
Data analyses were performed using the Statistical Package for Social Sciences for Windows v26.0.a Median scores were calculated for the 3 questions about the individual-specific value of participation: meaningfulness of participation, perceived strain, and meaningfulness of perceived strain. Spearman's correlation analyses were used to evaluate the correlation between the 3 questions of the participation activities on location using the 7-day average scores of each participant (≤0.30 is indicated as poor, 0.30-0.50 is fair, 0.50-0.80 is moderate, and ≥0.80 is very strong/perfect correlation
See the flow diagram (figure 2) for enrollment. In total, 72 of the 77 participants successfully completed the 7-day assessment period. Characteristics are displayed in table 2.
In total, 4958 segments were generated by the Whereabouts app (see table 3). Of these, a total of 3478 segments were assessed as participation on location and transport. It was not possible to distinguish activities at home; therefore, 39.4% automatically generated home-segments (<11-m radius) were excluded (see supplemental appendix S1, available online only at http://www.archives-pmr.org/).
Of the 3478 segments on location and transport, 85.6% of participation activities were labeled by the users using the predefined societal participation domains, 8.1% were described as “other,” and 6.3% of the labels were missing. Missing segments and other segments were analyzed based on the notes, aiming to classify the segment with a predefined domain. After analysis, independent recoding, and consensus (between IE and MO), 57.9% of these segments described as other or missing could be labeled under the predefined domains and 17.9% were classified as an incorrectly generated segments or were not filled in by users. Only 7.0% of participation activities were finally labeled as other; for example, funeral, information meeting, and religious activities or the transport segments running and by boat. The domains “in transit” (eg, train or bus stop) and “e-bike” were created by the researchers to label 17.2% of the other or missing participation activities.
The median duration of societal participation was 0.40 hours (range, 0.00-50.72 hours). For transport the total median duration was 0.15 hours (range, 0.00-13.65 hours). The societal participation and transport domains are summarized in table 4.
Table 4Amount of societal participation and its meaningfulness, perceived strain, and meaningfulness of the perceived strain over 7 days
Participation Domains
n (%)
Quantity in Hours, Median [Range]
n
Meaningfulness of Participation, Median [Range]
Perceived Strain, Median [Range]
Meaningfulness of Perceived Strain, Median [Range]
As shown in table 4, the individual-specific self-reported value of societal participation varied between the 3 questions from a median of 5 for perceived strain to 8 for overall meaningfulness and overall meaningfulness of the perceived strain. The median of perceived strain of each type of transport varied between 2 and 6, and the median meaningfulness of the perceived strain varied between the different domains, with e-bike as the lowest (5) and “other” domain as the highest (10).
We included 1302 participation activities on location within 7 days, with an average number of 19 participation activities per participant (range, 2-41 activities). We found a significant correlation between meaningfulness of participation and meaningfulness of perceived strain (r=0.857; P<.001); no significant correlation was found between meaningfulness and perceived strain or between perceived strain and meaningfulness of perceived strain (r=−0.152; P=.20 and r=−0.125; P=.28, respectively).
Figure 3 shows the median quantity, meaningfulness, perceived strain, and meaningfulness of strain for 2 participants. The figure illustrates interindividual differences in societal participation profiles as measured by the Whereabouts app.
Fig 3Overview of societal participation of 2 typical participants A and B. Quantity in median hours (blue) and median on a 1- to 10-point scale of the value constructs meaningfulness of participation (orange), perceived strain (gray), and meaningfulness of strain (yellow).
In total, 3.9% of segments were not labeled by users or generated inaccuracies.
Precision radius
Forty segments were generated with the predefined precision radius of ≥80 m, which means that the true location could not be accurately determined. After evaluation, 5 segments were not well generated (duration of ≤1 minute: n=2; no transportation according to the user: n=1; and not filled in by user for unknown reason: n=2).
Outliers
Verification of the 73 outlying segments (≥8 hours) showed that 25 segments seemed appropriate (eg, work/education), 6 segments were not well generated, and 42 segments were at home. After evaluating, 15 segments could be explained by the home circle with a radius of 11 m. No explanation could be found in the data for the other (mostly generated overnight) long-lasting segments.
Battery balance
Of the 240 generated segments when battery levels were ≤25%, 95.8% seemed accurate. The other segments were with a precision radius ≥80 (n=5), lasted ≥8 hours (n=2), were in the home circle (n=2), or were labeled as inaccurate (n=1).
Additional notes
Of the 571 notes made by the users, 54.6% were about inaccuracy of the generated segments, 45.0% were to specify activities at home, and 0.4% were filled in incorrectly by users.
User satisfaction
As showed in figure 4, a slight majority of participants were satisfied with the usability of the app (57.3%). The app gave a realistic overview of their daily activities (59.7%). More than half of the participants missed the possibility to specify participation activities at home, to add activities in more detail to make it more personal, or to make changes to correct mistakes (54.4%). In addition, 85.1% valued the question about perceived strain as important, 79.1% valued the question to label the participation activity as important, and 70.2% and 67.2% valued the other 2 questions about the meaningfulness and meaningfulness of perceived strain as meaningful, respectively. Furthermore, 66.2% would definitely or maybe use the app more often. They became more aware of which activities they conducted and how much effort the activities cost. Lastly, 77.9% definitely or probably want to share their societal participation overview with their physician or therapist for shared decision making about treatment goals.
Fig 4Perceived satisfaction (overall usability, n=68; realistic overview, n=67) and importance (Q1-Q4, n=67) about the Whereabouts app (%).
The Whereabouts app was developed to assess individual-specific meaningful societal participation in real time. In this study the usability of the Whereabouts app was tested in PwMS by assessing the effectiveness, efficiency, and satisfaction.
The Whereabouts app was shown to be effective. The app generates real-time type, frequency, and duration of different societal participation domains and contains individual-specific value constructs.
The ICF was used to define the participation domains of the Whereabouts app.
The ICF subdomain “caring for others” was taken as a main domain and “health care utilization” was added, because we suggest that these could be important domains in rehabilitation medicine. The literature has shown that societal participation is difficult to measure due to the diversity of subdomains and differences in relevance for individuals.
In the Whereabouts app, participants were able to address their own unique participation profiles, allowing differences in participation within and between persons. Participation activities in the other domain helped us to assess whether important subdomains were overlooked in the development of the app. Almost 60% of participation activities labeled as other could be labeled by predefined domains, suggesting that the main domains were not immediately recognized and are not that specific, as suggested by the users. Besides individual differences, technical aspects and the appearance of new domains contribute to the explanation of the addressed other activities. The domain “in transit” was created because users defined participation activities as “train or bus stop,” because the app is not able to combine domains consisting of location and transport segments. The e-bike was often mentioned and might require separate labeling. It could be important for setting treatment goals. Therefore, it is likely important to adapt the app to select activities that are more specific and personal and to give users access to more examples of the participation domains in a manual.
The value of societal participation is an important construct to measure.
Results showed a statistically significant strong correlation between the questions about meaningfulness of participation and meaningfulness of perceived strain. Users’ feedback showed that the question about meaningfulness of perceived strain was the least valuable. Taken together, these findings suggest that this question is redundant and could be excluded to improve the effectiveness.
Efficiency
Only a small percentage of the segments were not complete or inaccurate, suggesting that the app is efficient in tracking locations and transport. To maintain the efficiency, continual testing and upgrading of the app is needed in newer versions because of ongoing modifications in mobile phone software.
Software engineering issues for mobile application development.
in: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research—FoSER ’10. Association for Computing Machinery,
Santa Fe, NM2010: 397-400
There is no indication of selective inefficiency due to duration of segments, accuracy of GPS signals, or battery level, and no additional notes were made by users.
User satisfaction
A majority of the participants were satisfied with the usability of this first version. Suggested elements for improvement were options to (1) change previous answers or to delete an inaccurate segment, (2) specify activities at home, and (3) add or specify activities in more detail to make it more personal. These recommendations will be integrated in a next version to further improve user satisfaction.
For clinical implementation, the option to generate a graphical overview of the societal participation for the user (as illustrated in figure 3) and the ability for the user to share this with health care providers as a therapy indicator or as an outcome measure would increase the usability of the app. Owing to large interindividual differences in participation, individual-specific scores as well as intra-individual changes in participation scores over time are more useful for a tailored approach, self-management, and individual goal setting.
Further research is needed in rehabilitation settings to assess the usability of the Whereabouts app specifically by people with treatment goals in societal participation. Studies show that acceptance of such apps is expected to increase when these are integrated into clinical treatment plans.
In this study the ISO-9241 standard was used, which includes 3 usability constructs. According to Abran et al, this standard does not address the learnability characteristic (ie, capability of the software product to enable the user to learn its application) as recommended by the majority of standards and experts on usability.
Although learnability was not specifically assessed in this study, users’ feedback suggested that the app was easy to handle with little instruction. Nevertheless, special attention will be paid to make domains more recognizable because participation activities were unnecessary labeled as other.
Furthermore, the ISO-9241 standard does not address security aspects, which are considered very significant by experts in this field. Obviously, it is important to consider privacy and security aspects because of the sensitive nature of location data. We monitored these aspects in our study carefully. This study was approved by The Medical Ethical Committee of Amsterdam UMC, location VUmc, a Privacy Impact Assessment has been conducted by the Amsterdam UMC, and the digital biomarker developer Orikami Digital Health Products is compliant with General Data Protection Regulation under ISO-27001 and NEN-7519 industry (medical) information security standards.
To facilitate implementation, it is important to regularly evaluate user satisfaction.
In this study, we used a self-developed evaluation form. In a next step, qualitative methods—for example, focus groups, interviews, or existing questionnaires, such as the System Usability Scale—would be useful to clarify the experiences with the app in more detail.
although in our study more PwMS with mild disability were included. Although the application seems easy to handle when people are capable of using a smartphone (eg, automatic GPS measurement, only a few answers are necessary without many clicks), it is important to look more closely at usability for people with visual problems, problems with hand function, or cognition difficulties, for example. Because disability influences societal participation,
it is also important to study the generalizability of the Whereabouts app in other populations with disabilities.
This first version of the app is limited to assessing only participation outside, though participation often takes place inside home (eg, online shopping, social contact, working at home). If a full participation profile is needed, participation both on location and at home needs to be assessed. Logbooks and technical improvements in smartphones and the Whereabouts app can help to fill this gap.
Future plans
This first study about usability showed that improvements and more research for successful use in the future are needed. In a next step we will investigate the validity of the collected data at the group level; for example, by comparing these data with existing participation questionnaires and known subgroup comparisons. Furthermore, possibilities for and validity of summary measures at the group level will be investigated. The Whereabouts app will be provided for use by PwMS as well as for multidisciplinary collaborative use among rehabilitation professionals to facilitate the assessment of restrictions and changes in individual-specific meaningful societal participation profiles. It is possible that participation profiles, including meaningfulness, can change over time if they are measured over a longer period of time or in different seasons. In a clinical setting with individual patients, it is therefore important to choose the most reasonable assessment period and decide on the circumstances during this period. We intend to use the results from the present study to integrate the Whereabouts data with physiological measurements, specific questionnaires, goal setting, and, potentially, a data sharing and communication platform. These combined results can subsequently be presented through a dashboard available to the patient and their rehabilitation professional. With this data playback, results can be linked to treatment goals. In addition, evidence-based treatment advice can be integrated. This app also has the potential to be the next step in self-management programs, in shared decision making, and in personalized, evidence-based rehabilitation medicine and outcome assessment that aim to optimize and prevent decline of individual-specific meaningful societal participation.
Conclusions
The newly developed Whereabouts app has been shown to be usable to register real-time individual participation activities as well as the appreciation of these activities. Overall, users rated the app as an added value and were satisfied with the use of this first version. Improvements are recommended, such as the possibility to specify participation activities and to generate a graphic overview.
Acknowledgments
We thank Stijn Albers van der Linden from Orikami Digital Health Products for his role in the development of the application and Nienke Heida for conducting home visits.
in: Turner T. Szwillus G. Czerwinski M. Paternò F. Pemberton S. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems–CHI ’00. The Hague. Association for Computing Machinery,
The Netherlands2000: 345-352
Software engineering issues for mobile application development.
in: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research—FoSER ’10. Association for Computing Machinery,
Santa Fe, NM2010: 397-400
Supported by The Dutch MS Research Foundation (16-918 MS).
Disclosures: Inez Wijnands and Frank Dorssers are employed by Orikami Digital Health Products. A consortium agreement, including a financial agreement, was made for the development of the app between VUmc and Orikami.
Mean ± SD.
00263-5&id=gr1.jpg){kind=link}
00263-5&id=gr2.jpg){kind=link}
00263-5&id=gr3.jpg){kind=link}
00263-5&id=gr4.jpg){kind=link}