Quality of a Supporting Mobile App for Rheumatic Patients: Patient-Based Assessment Using the User Version of the Mobile Application Scale (uMARS)

Introduction: Mobile applications promise to improve current health care. However, current mobile app quality ratings are mostly physician-based. The aim of this study was (1) to assess the quality of the self-management app Rheuma Auszeit using the validated uMARS (User Version of the Mobile App Rating Scale) app quality assessment tool and (2) to evaluate the association between uMARS scores and patients' characteristics.

Materials and Methods: Consecutive patients with rheumatoid arthritis, psoriatic arthritis and spondyloarthritis were seen at the rheumatology clinic at university hospital Erlangen, Germany. They were asked to test Rheuma Auszeit, evaluate its quality using uMARS and complete a paper-based survey evaluating the individual preferences, attitudes and ehealth literacy. The association between uMARS scores and patients' characteristics was further explored.

Results: Between December 2018 and January 2019, a total of 126 patients evaluated Rheuma Auszeit using uMARS and filled out the paper-based survey. The median uMARS score was 3.9, IQR 0.7. Functionality was the domain with the highest rating (median 4.8, IQR 0.8), followed by aesthetics (median 4.0, IQR 0.7), information (median 3.5, IQR 0.8), and engagement (median 3.2, IQR 1.0). Subjective quality was average (median 3.0, IQR 1.0). The lowest scoring individual item was customization with a median of 2.5/5. Lower functionality scores were reported among older female rheumatic patients (P < 0.004). Older male rheumatic patients reported a higher subjective quality score (P < 0.024). Perceived disease activity and disease duration did not significantly correlate with any uMARS subdomain scores. eHealth literacy significantly correlated with functionality uMARS subdomain ratings (Rho = 0.18; P < 0.042). Preferred time of app usage significantly correlated with engagement (Rho = 0.20; P < 0.024), functionality (Rho = 0.19; P < 0.029), total uMARS score (Rho = 0.21; P < 0.017) and subjective quality score (Rho = 0.21; P < 0.017). The vast majority of rheumatic patients would consider recommending Rheuma Auszeit to other patients (117/126; 92.9%).

Conclusion: Rheuma Auszeit was well-accepted by German patients suffering from rheumatoid arthritis, psoriatic arthritis and ankylosing spondyloarthritis. Lacking customization could lead to low app compliance and should be improved. Lower functionality scores among older female rheumatic patients highlight the need for patient education. The study underlines the potential and feasibility of therapeutic complementary digital solutions in rheumatology.

Introduction

Rheumatology encompasses a variety of chronic inflammatory diseases that require immunosuppressive treatment. Despite more effective medical treatment becoming available to patients, many patients are unable to reach clinical remission. An integrative, holistic treatment approach might improve patients' quality of life. Due to the wide adoption of smartphones (1), medical apps could become an effective tool (2–5) to promote predictive, preventive, personalized and participatory (“P4”) medicine (6).

The European Alliance of Associations for Rheumatology (EULAR) anticipated the considerable potential of mobile apps to aid rheumatic patient disease self-management and recently published guidelines (3) to support the implementation into clinical care. Medical apps are already being used in clinical routine by an increasing number of rheumatologists (7). However, little knowledge of the quality of existing apps (8) and evidence (9) supporting their effectiveness and acceptance currently prevents widely adoption (10). In a recent physician-based AppStore analysis (4), Rheuma Auszeit received the highest rating among sixteen identified apps meeting inclusion criteria. Notably, Rheuma Auszeit was the only app, mainly developed by rheumatic patients (German League against Rheumatism). Most of the apps provided symptom tracking, whereas Rheuma Auszeit delivers video and audio instructions for mental and physical exercises.

Mobile apps created for patient use are currently primarily assessed exclusively by physicians (4, 5), yet it has been repeatedly stressed to include end-user (patient) evaluations (1, 4, 11–13). The perceived app quality is subjective and could significantly differ between patients and physicians, similar to perceived disease activity (14). To enable a user-centric app evaluation, the user version of the Mobile App Rating Scale (uMARS) has been created (15). It assesses the dimensions of engagement, functionality, aesthetics, information, and subjective quality on 5-point scales and has been used to evaluate the usability of dengue fever apps (16), mental health apps (17) and wearable sensor-based biofeedback systems (18). User-centered app evaluations are crucial to improve app design, effectiveness and importantly app adherence (4). In order to maximize app effectiveness and long-term usage, continuous improvement and personalization are necessary (1, 4, 9, 12, 13).

The aim of this study was to assess the quality of Rheuma Auszeit by rheumatic patients using uMARS and to explore associations between uMARS scores and patient characteristics.

Materials and Methods

Consecutive patients with rheumatoid arthritis (RA), psoriatic arthritis (PsA) and spondyloarthritis (SpA) were recruited between December 2018 and January 2019 at the rheumatology clinic at University hospital Erlangen, Germany. The study was approved by the local Ethics committee (No. 418-18B) and conducted referring to good clinical practice. All patients provided informed consent and were then asked to (1) evaluate the app Rheuma Auszeit using uMARS and to (2) complete a paper-based survey evaluating the individual preferences, attitudes and eHealth literacy. The association between uMARS scores and patient characteristics were further explored.

Rheuma Auszeit

Rheuma Auszeit (Figure 1) has been developed by the German RMD patient organization Rheuma-Liga and is available to download for free in the German App Store and Google Play store. It offers rheumatic patients, irrespective of the exact rheumatic disease, practical self-management recommendations that can immediately be applied to support patients coping with rheumatic pain. The app consists of audio, images and videos that should guide patients. Exercises manuals include muscle relaxation, self-massage, mind journeys, movement training and cold/hot treatments. There is no predetermined order of exercises and patients are free to select their preferred exercises. Importantly, symptom tracking/monitoring offered by various other apps (4) is not offered by this app. All material can be downloaded to support offline usage.

FIGURE 1

Figure 1. Screenshots of the Rheuma Auszeit app: (A) logo, (B) main menu including the following options: stress relief, imagination stimulator, pick-me-up, cool down & warm up, power donor, feedback. Images provided by the developer, German League against Rheumatism.

App Evaluation Using uMARS

Local study personnel explained the goal and functions of the app and gave a short demonstration. As recommended by the uMARS developers (15), patients tested Rheuma Auszeit for at least 10 min using tablets (IPads). Patients were then asked to evaluate the app quality employing the paper version of the German version of the uMARS (15), which is a simplified user version of the MARS (19, 20) including 20 items instead of the original 23. MARS has been used to evaluate the quality of a multitude of apps and its construct validity, reliability and objectivity has been recently validated in a multicenter study (19). Both rating scales assess the five dimensions: engagement, functionality, aesthetics, information, and subjective quality on 5-point scales. Instead of the original 7 information quality MARS items, uMARS consists of four information quality items: quality of information, quantity of information, visual information, credibility of source. The subjective quality dimension is based on four questions also using 5-point scales: willingness to recommend the app, anticipated app usage frequency, willingness to pay for the app, and overall rating.

Patient Survey

Besides the app evaluation, patients were asked to complete a paper-based survey regarding individual sociodemographic details, mhealth preferences, attitudes and eHealth literacy. Patients' eHealth literacy was measured using the validated German version (21) of the eHealth Literacy Scale (eHEALS) (22). It is based on a 5-point Likert scale and includes eight statements concerning self-perceived eHealth literacy. We previously published exact survey details and results from a larger patient collective (1). In this study survey data was only used from patients that evaluated Rheuma Auszeit.

Statistical Analysis

The statistical analysis was split into three distinct steps:

(i) Patient characteristics were summarized using medians, interquartile ranges, means, standard deviations, counts, and percentages as appropriate. Descriptive statistics were computed for describing distributions of variables related to uMars score and subjective quality among patients. In details, boxplots and distribution histograms were used to summarize scores distributions from the whole population regarding engagement, functionality, aesthetics, information, total score and subjective quality variables. Among these variables, (1) engagement, (2) functionality, (3) aesthetics and (4) information variables resulted from a combination of several other variables averaged together, respectively (1) entertainment, interest, customization, interactivity, target group, (2) performance, ease of use, navigation, gestural design (3) layout, graphics, visual appeal, (4) quality of information, quantity of information, visual information and credibility of source. Thus, supplementary boxplots were computed for summarizing the distribution of each of these combination variables. Finally, boxplots computed from the whole population for describing the main variables, i.e., engagement, functionality, aesthetics, information, total score and subjective quality, were visually compared with subgroups of the patient populations related to gender and disease. As such, violin plots were used to compare the whole patient population with (1) axial spondyloarthrisis, (2) psoriatic arthritis, (3) rheumatoid arthritis. Similarly, violin plots were also used to compare the whole patient population with (1) female and (2) male patients.

(ii) Association between main variables of interest were calculated by means of Spearman's correlation test. Spearman's rho and its corresponding p-value were calculated for correlating each mobile app rating scales (i.e., engagement, functionality, aesthetics, information, total score and subjective quality variables) with variables related to (1) sociodemographics (population age, female age, male age), (2) health characteristics (disease duration, disease activity), (3) mHealth characteristics (ehealth literacy, prefered time for app usage) and (4) the importance of app characteristics (interactivity, design, usability, data security). Correlations were calculated from the whole patient population and for each disease subgroup, separately.

(iii) Each of the variables described above were compared according to (1) gender (female vs. male patients) and to disease (axial spondyloarthritis vs. psoriatic arthritis vs. rheumatoid arthritis patients) using Mann-whitney comparative tests. Total score and Subjective quality variables were compared for (1) the whole population, (2) female, (3) male, (4) axial spondyloarthritis, (5) psoriatic arthritis, and (6) rheumatoid arthritis patients, separately.

The level of significance was set as p < 0.05 in all statistical tests. All statistical calculations were completed using the R software environment (version 3.1.0; R Foundation for Statistical Computing, Vienna, Austria).

Results

Patient Characteristics

Patient demographic information is displayed in Table 1. In total, 126 rheumatic patients, of which 76 (60.3%) were female, completed the study. This population consisted of 59 (46.8%) RA, 27 (21.4%) SpA, and 40 (31.7%) PsA patients. Mean disease duration was 7.8 years (SD 7.2) and mean patient global disease activity was 4.1 (SD 2.4).

TABLE 1

Table 1. Demographic and health characteristics.

Table 2 displays the patient mHealth attitude and current usage. The majority of patients used smartphones on a regular basis, almost half of the patients regulary used social media and only a minority regularly used activity trackers. Mean eHealth literacy using eHEALS was (26.1/40; SD 7.1). Most patients believed that medical apps are helpful for their health (89/126; 70.6%) and were willing to use them in the future (83/126; 65.9%), preferably for a maximum of 5-15 min (43/126; 34.1%). Nearly all patients (125/126; 99.2%) were willing to share app data for research purposes. The majority of patients 86/126; 68.3%) was previously looking for treatment options on the Internet and was interested in an app including physical exercises and stress reduction activities (94/126; 74.6%). Most patients wanted official app recommendations from the national society of rheumatology (98/126; 77.8%) and only a small minority was aware of useful rheumatology apps/websites (10/126; 7.9%) or previously participated in an online health program (2/126; 1.6%). Usability and data security were rated as more essential app features (10) than design (5) and interactivity (5) by patients.

TABLE 2

Table 2. Patient mHealth attitude and usage.

Rheuma Auszeit uMARS Rating

Figure 2 shows the uMARS subdomain ratings, total score and subjective quality according to disease group. The overall median uMARS score was 3.9/5, IQR 0.7. Functionality was the domain with the highest rating (median 4.8, IQR 0.8), followed by aesthetics (median 4.0, IQR 0.7, information (median 3.5, IQR 0.8), and engagement (median 3.2, IQR 1.0). Engagement scores were significantly lower for SpA patients (P < 0.047) and PsA patients (P < 0.016) compared to RA. Subjective quality was average (median 3.0, IQR 1.0) and was significantly lower in SpA compared to PsA patients (P < 0.013).

FIGURE 2

Figure 2. Violin plots of uMARS subdomains, total score and subjective quality according to disease group.

Figure 3 shows no significant gender-based differences regarding uMARS subdomain ratings, total score and subjective quality. Figure 4 presents the single uMARS items of the four subdomains, uMARS total score and subjective quality. The highest individual item scores were the functionality subdomain items (performance, ease of use, navigation and gestural design), each with a median of 5/5. The lowest scoring item was customization with a median of 2.5/5.

FIGURE 3

Figure 3. Violin plots of uMARS subdomains, total score and subjective quality according to sex.

FIGURE 4

Figure 4. Box plots of single uMARS subdomain items, total score and subjective quality.

Table 3 lists the subjective app quality uMARS item ratings. Only a minority of patients would not at all recommend Rheuma Auszeit (9/126; 7.1%) and most patients (61/126; 48.4%) would maybe recommend it. Similarly, only 14/126 (11.1%) patients think that they would never use Rheuma Auszeit. 10/126 (7.9%) patients would definitely pay for this app. The majority of patients assigned an average rating (68/126; 54.0%).

TABLE 3

Table 3. Subjective app quality uMARS item ratings.

Supplementary Table 1 displays the correlations between uMARS ratings, users' socio-demographics, health characteristics and eHealth literacy. We observed a significant correlation with increasing age and lower functionality uMARS subdomain ratings for female patients (Rho = −0.33; P < 0.004). Increasing age correlated positively with subjective quality (Rho = 0.32; P < 0.024) and total uMARS score (Rho = 0.25; P < 0.083) for male patients.

Perceived disease activity and disease duration did not significantly correlate with any uMARS items. eHealth literacy significantly correlated with functionality uMARS subdomain ratings (Rho = 0.18; P < 0.042). Preferred time of app usage significantly correlated with engagement (Rho = 0.20; P < 0.024), functionality (Rho = 0.19; P < 0.029), total uMARS score (Rho = 0.21; P < 0.017) and subjective quality score (Rho = 0.21; P < 0.017). Importance of usability correlated with functionality scores (Rho = 0.18; P < 0.048) and total scores (Rho = 0.19; P < 0.035).

Discussion

In a previous German app store analysis (4), Rheuma Auszeit received the highest physician-based MARS rating (4.2/5). It was however unclear if rheumatic patients share this positive perception. This study now complements the previous evaluation by presenting a detailed patient perspective, as demanded in the published EULAR recommendations (3). To our knowledge, Rheuma Auszeit is the first German rheumatology app that has been tested by both, physicians and patients. Our study provides valuable patient insights regarding the app's usability in terms of its functions, engagement, design, information, and subjective qualities.

This study shows that Rheuma Auszeit is well-accepted among rheumatic patients. In particular, the app received high functionality and aesthetics ratings. Most patients stated that they would use the app 3–50 times (91; 72.2%) during the next 12 months and only a minority of patients (14/126; 11.1%) would not use the app at all. We did not observe significant correlations between disease activity, disease duration and total uMARS score.

The overall uMARS score was 3.9/5 compared to the physician-based MARS score (4.2/5) (4). Similar to the previous physician-based MARS analysis (4), subjective quality was significantly (P < 0.0001) lower compared to the overall uMARS score. The lower subjective ratings could be due to a general German conservativeness. Importantly our patient results are in line with physician evaluations rating the engagement subdomain as the worst uMARS subdomain (4) and showing no significant gender rating differences. We believe that different disease specific app expectations could have caused the differences regarding the uMARS engagement subsection. For example SpA patients are likely to be more engaged if spine associated features are addressed in the app.

Patients were reluctant to pay for the app and actively recommend it to other patients. Most patients would welcome such app recommendations from the national society of rheumatology. The German digital healthcare act (Digital Versorgung Gesetz-DVG) allows to partially overcome these implementation barriers, as physicians can now prescribe designated medical apps to patients, which are then reimbursed by insurance companies.

Rheuma Auszeit (translated: Rheuma Timeout) is unique, as it was the only app in a recent review mainly developed by rheumatic patients offering coping techniques (4). The top two app features rheumatic patients named in a previous survey were information (medication and disease) and exercises to reduce stress and pain (1). Rheuma Auszeit addresses the latter. Most other previously identified German rheumatic apps focus on symptom tracking (4), however offer no techniques to reduce stress and pain. Lalloo et al. showed that symptom tracking alone can lead to a clinically meaningful reduction in pain intensity in a population with juvenile idiopathic arthritis (23). On the other hand, Seppen et al. showed that pain positively mediated app usage in RA patients but some patients identified symptom tracking (reminding patients of their disease) as a barrier for app usage (24). This suggests that symptom tracking frequency should be discussed with patients via shared-decision-making (SDM). Shaw et al. recently showed that patient-reported outcome results should be discussed during face-to-face time to enhance the patient-provider interaction (25).

We believe that rheumatic patients need to receive proper instructions on how to use medical apps similar to other medical products. This personalized training will likely improve app adherence and effectiveness, however will most likely not be carried out by the treating physician due to the pressing time constraints. The lack of information on available tools and supporting evidence are main barriers that delay wide adoption. Our study showed lower functionality scores among older female patients (P < 0.004) and higher subjective quality scores among older male patients (P < 0.024). This suggests that not all rheumatic patients might benefit equally from mHealth solutions and instructions need to be personalized as well.

The high overall uMARS score and number of rheumatic patients regularly using smartphones imply that a longitudinal clinical study is feasible. The low customization score suggests however that such a trial will likely result in low app adherence, especially since app retention in general is an imminent mHealth barrier (26) also present in rheumatology (24, 27). In our opinion, customizability should be improved by gathering additional qualitative feedback from rheumatic patients before initiating a longitudinal clinical study evaluating clinical benefits of the app.

This study has some limitations. The analysis was solely based on quantitative feedback from a single evaluation tool based on short app usage. No sample size calculation was carried out. Qualitative feedback, similar to Herbuela et al. (16), would have added valuable depth and details to our analysis and are planned to be carried out. Furthermore, the disease selection limits the generalizability to other rheumatic diseases and only patient-reported items were collected in the study. We received a lot of positive feedback especially from older patients that would not have come in contact with Rheuma Auszeit otherwise. This feedback was however not properly measured. Furthermore, Rheuma Auszeit is currently only available in German, limiting the potential international impact. The German uMARS version which we used has not been published yet, however the items are identical to the published German MARS translation (28). The System Usability Scale (SUS) (24, 29) and Net Promoter Score (NPS) (24) are increasingly being used in app evaluation studies and could have enhanced our work.

Conclusion

Rheuma Auszeit is well-accepted among patients suffering from different rheumatic diseases, different ages and digital competencies. Physicians and other health care professional should be encouraged to integrate mHealth into their clinical routine. Lacking customizability of Rheuma Auszeit diminished the total uMARS score and will likely limit long-term app usage by patients.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by ethics committee of the Friedrich-Alexander University Erlangen-Nürnberg (FAU); (No. 418-18B). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

AL and JK wrote the draft manuscript. AL, JK, TA, and NV performed the statistical analysis. All authors reviewed the draft and provided comments for changes, approved the final manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors thank all participating patients and the whole team of Medizinische Klinik 3 for their support of this project. The present work was performed in fulfillment of the requirements for obtaining the degree Dr. med for the first author, AL, and is part of the PhD thesis of the last author, JK (AGEIS, Université Grenoble Alpes, Grenoble, France).

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2021.715345/full#supplementary-material

References

1. Knitza J, Simon D, Lambrecht A, Raab C, Tascilar K, Hagen M, et al. Mobile health usage, preferences, barriers, and ehealth literacy in rheumatology: patient survey study. JMIR mHealth uHealth. (2020) 8:e19661. doi: 10.2196/19661

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Iribarren SJ, Akande TO, Kamp KJ, Barry D, Kader YG, Suelzer E. Effectiveness of mobile apps to promote health and manage disease: systematic review and meta-analysis of randomized controlled trials. JMIR mHealth uHealth. (2021) 9:e21563. doi: 10.2196/21563

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Najm A, Nikiphorou E, Kostine M, Richez C, Pauling JD, Finckh A, et al. EULAR points to consider for the development, evaluation and implementation of mobile health applications aiding self-management in people living with rheumatic and musculoskeletal diseases. RMD Open. (2019) 5:e001014. doi: 10.1136/rmdopen-2019-001014

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Knitza J, Tascilar K, Messner E-M, Meyer M, Vossen D, Pulla A, et al. German mobile apps in rheumatology: review and analysis using the mobile application rating scale (MARS). JMIR mHealth uHealth. (2019) 7:e14991. doi: 10.2196/14991

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Dantas LO, Carvalho C, Prando BC, McAlindon TE, da Silva Serrão PRM. Mobile health technologies for the management of rheumatic diseases: a systematic review of online stores in Brazil. Clin Rheumatol. (2021) 40:2601–9. doi: 10.1007/s10067-020-05561-y

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Flores M, Glusman G, Brogaard K, Price ND, Hood L. P4 medicine: how systems medicine will transform the healthcare sector and society. Per Med. (2013) 10:565–76. doi: 10.2217/pme.13.57

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Knitza J, Vossen D, Geffken I, Krusche M, Meyer M, Sewerin P, et al. Nutzung von medizin-apps und online-plattformen unter deutschen rheumatologen. Z Rheumatol. (2019) 78:839–46. doi: 10.1007/s00393-018-0578-3

CrossRef Full Text | Google Scholar

8. Krusche M, Klemm P, Grahammer M, Mucke J, Vossen D, Kleyer A, et al. Acceptance, usage, and barriers of electronic patient-reported outcomes among German rheumatologists: survey study. JMIR mHealth uHealth. (2020) 8:e18117. doi: 10.2196/18117

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Kernder A, Morf H, Klemm P, Vossen D, Haase I, Mucke J, et al. Digital rheumatology in the era of COVID-19: results of a national patient and physician survey. RMD Open. (2021) 7:e001548. doi: 10.1136/rmdopen-2020-001548

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Mühlensiepen F, Kurkowski S, Krusche M, Mucke J, Prill R, Heinze M, et al. Digital health transition in rheumatology: a qualitative study. Int J Environ Res Public Health. (2021) 18:2636. doi: 10.3390/ijerph18052636

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Baptista S, Oldenburg B, O'Neil A. Response to “development and validation of the user version of the mobile application rating scale (uMARS).” JMIR mHealth uHealth. (2017) 5:e6419. doi: 10.2196/mhealth.6419

CrossRef Full Text | Google Scholar

12. Grainger R, Townsley H, White B, Langlotz T, Taylor WJ. Apps for people with rheumatoid arthritis to monitor their disease activity: a review of apps for best practice and quality. JMIR mHealth uHealth. (2017) 5:e7. doi: 10.2196/mhealth.6956

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Serlachius A, Schache K, Kieser A, Arroll B, Petrie K, Dalbeth N. Association between user engagement of a mobile health app for gout and improvements in self-care behaviors: randomized controlled trial. JMIR mHealth uHealth. (2019) 7:e15021. doi: 10.2196/15021

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Studenic P, Radner H, Smolen JS, Aletaha D. Discrepancies between patients and physicians in their perceptions of rheumatoid arthritis disease activity. Arthritis Rheum. (2012) 64:2814–23. doi: 10.1002/art.34543

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the mobile application rating scale (uMARS). JMIR mHealth uHealth. (2016) 4:e72. doi: 10.2196/mhealth.5849

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Herbuela VRDM, Karita T, Carvajal TM, Ho HT, Lorena JMO, Regalado RA, et al. Early detection of dengue fever outbreaks using a surveillance app (Mozzify): cross-sectional mixed methods usability study. JMIR Public Health Surveill. (2021) 7:e29795. doi: 10.2196/preprints.29795

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Strodl E, Shakespeare-Finch J, Alichniewicz KK, Brown K, Quinn C, Hides L, et al. Clinicians' perceptions of PTSD coach australia. Int Interv. (2020) 21:100333. doi: 10.1016/j.invent.2020.100333

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Argent R, Slevin P, Bevilacqua A, Neligan M, Daly A, Caulfield B. Wearable sensor-based exercise biofeedback for orthopaedic rehabilitation: a mixed methods user evaluation of a prototype system. Sensors. (2019) 19:432. doi: 10.3390/s19020432

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Terhorst Y, Philippi P, Sander LB, Schultchen D, Paganini S, Bardus M, et al. Validation of the mobile application rating scale (MARS). PLoS ONE. (2020) 15:e0241480. doi: 10.1371/journal.pone.0241480

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth. (2015) 3:e3422. doi: 10.2196/mhealth.3422

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Soellner R, Huber S, Reder M. The concept of ehealth literacy and its measurement. J Media Psychol. (2014) 26:29–38. doi: 10.1027/1864-1105/a000104

CrossRef Full Text | Google Scholar

22. Norman CD, Skinner HA. eHEALS: the ehealth literacy scale. J Med Int Res. (2006) 8:e507. doi: 10.2196/jmir.8.4.e27

CrossRef Full Text | Google Scholar

23. Lalloo C, Harris LR, Hundert AS, Berard R, Cafazzo J, Connelly M, et al. The iCanCope pain self-management application for adolescents with juvenile idiopathic arthritis: a pilot randomized controlled trial. Rheumatology. (2021) 60:196–206. doi: 10.1093/rheumatology/keaa178

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Seppen BF, Wiegel J, L'ami MJ, Duarte Dos Santos Rico S, Catarinella FS, Turkstra F, et al. Feasibility of self-monitoring rheumatoid arthritis with a smartphone app: results of two mixed-methods pilot studies. JMIR Form Res. (2020) 4:e20165. doi: 10.2196/20165

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Shaw Y, Courvoisier DS, Scherer A, Ciurea A, Lehmann T, Jaeger VK, et al. Impact of assessing patient-reported outcomes with mobile apps on patient-provider interaction. RMD Open. (2021) 7:e001566. doi: 10.1136/rmdopen-2021-001566

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Brindal E, Hendrie GA, Freyne J, Noakes M. Incorporating a static versus supportive mobile phone app into a partial meal replacement program with face-to-face support: randomized controlled trial. JMIR mhealth uhealth. (2018) 6:e7796. doi: 10.2196/mhealth.7796

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Colls J, Lee YC, Xu C, Corrigan C, Lu F, Marquez-Grap G, et al. Patient adherence with a smartphone app for patient-reported outcomes in rheumatoid arthritis. Rheumatology. (2021) 60:108–12. doi: 10.1093/rheumatology/keaa202

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Messner E-M, Terhorst Y, Barke A, Baumeister H, Stoyanov S, Hides L, et al. The German version of the mobile app rating scale (MARS-G): development and validation study. JMIR mHealth uHealth. (2020) 8:e14479. doi: 10.2196/14479

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Grainger R, Townsley HR, Ferguson CA, Riley FE, Langlotz T, Taylor WJ. Patient and clinician views on an app for rheumatoid arthritis disease monitoring: function, implementation and implications. Int J Rheum Dis. (2020) 23:813–27. doi: 10.1111/1756-185X.13850

PubMed Abstract | CrossRef Full Text | Google Scholar