Jennifer Kircher1
Walter Swoboda
Felix Holl- 1DigiHealth Institute, Neu-Ulm University of Applied Sciences, Neu-Ulm, Germany
- 2Institute for Medical Information Processing, Biometry, and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany
Cardiovascular disease is one of the leading causes of death worldwide. Scarce resources and rising costs are pushing healthcare systems to their limits. There is an urgency to develop, optimize and evaluate technologies that provide more effective care for patients. Modern technologies, such as mobile health (mHealth) applications, can provide relief as a key strategy. To integrate digital interventions into care structures, a detailed impact assessment of all professional mHealth applications is needed. The aim of this study is to analyze the standardized tools used in the field of cardiovascular disease. The results show that questionnaires, usage logs, and key indicators are predominantly used. Although the identified mHealth interventions are specific to cardiovascular disease and thus require particular questions to evaluate apps, the user readiness, usability, or quality of life criteria are non-specific. Therefore, the results contribute to understanding how different mHealth interventions can be assessed, categorized, evaluated, and accepted.
1. Introduction
Studies such as Neumann et al. (1) have examined the growing number of digital health offerings, including mobile health (mHealth) apps, and expect them to double by 2025. Given the rapidly evolving market for digital health, there is an urgency to evaluate professional mHealth applications for their impact assessments. Such assessments will help determine whether the data and treatment outcomes are valid enough to provide quality care (2). Currently, there is no international consensus on standards for assessing health apps. Existing evaluation frameworks, such as the American Psychiatric Association app evaluation model, stand out in their flexibility of approach. However, this has also led to a demand for a more applied approach that provides more concrete information to users (3). An evaluation framework that identifies and instrumentalizes various exemplary methods and tools is lacking.
According to Kvedar et al. (4), developing, optimizing, and evaluating technologies that provide more effective care for patients is needed. mHealth interventions have great potential to support the treatment of cardiovascular diseases (CVD), which presents a considerable burden of diseases globally. In Germany, for example, 331,211 deaths were related to diseases of the cardiovascular system in 2019 (5). Earlier statistics from 2015 put the highest disease-specific illness costs in the German healthcare system at around 46.4 billion euros, caused by cardiovascular diseases (6). mHealth has the potential to improve CVD treatment by providing more personalized and timely care, supporting patient self-management, and encouraging healthy behaviors. Elements that mHealth applications can support are remote monitoring, patient education and self-management, as well as behavior change (7–9). However, developing applications for CVD comes with some unique challenges and obstacles compared to other disease-related apps. CVD is a complex condition that encompasses a range of different diseases and risk factors. Developing an app that addresses all aspects of CVD, from prevention to diagnosis to treatment, can be challenging (10). There is also a great variability in patient needs, as patients with CVD can have different needs and preferences depending on their specific condition and individual circumstances. Developing an app that is personalized and adaptable to different patient needs can be difficult (11).
The aim of this study is to identify which standardized tools are already used today for a comprehensive evaluation of mHealth applications in the field of cardiovascular disease. The resulting assessment, categorization, and evaluation findings will inform the development of an applicable evaluation framework for CVD mHealth interventions as a recommended course of action.
2. Materials and methods
To generate an initial impression of existing evaluation methods for assessing mHealth applications, a preliminary study has been conducted as part of a scoping review for CVD. The study “methods used to evaluate mHealth applications for cardiovascular disease: a quasi-systematic scoping review” (12) was published in the International Journal of Environmental Research and Public Health in 2021. The 38 studies already identified were narrowed down for the present study based on further exclusion criteria, and 37 studies (13–49) formed the starting point for this study. While the initial study provided an overview of all evaluation tools used, this study specifically looks at standardized tools that have been used in the evaluation of CVD mHealth applications and investigates their characteristics and possible shortcomings. Inclusion and exclusion criteria, the search strategy, and literature selection (including a table with the extracted data) can be found in the Supplementary material. These steps were done following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) scheme (50).
We categorized the identified tools into 8 application areas and filtered by tool type. We derived the categories through thematic analysis (51). In addition, we subdivided the mHealth application according to their intervention type. Treatments performed only by using an app belong to the “mHealth app” type. When multiple devices or technical components, such as an app, an ECG (electrocardiogram), or a smartwatch, are used, the studies fall into the “mHealth system” group. Under “mHealth text messaging” those studies are assigned whose intervention is based exclusively on text messages, in particular “short message service”(SMS)-based messages. Following the categorization, we investigated the characteristics and shortcomings of the tools.
3. Results
3.1. Characteristics
Fourthy-eight evaluation instruments have been identified from the 37 studies. Of the 48 methods, questionnaires (n = 29, 60%) and economic measures such as key performance indicators (n = 9, 18%) are predominantly used as evaluation tools. All instruments have been used a total of 122 times. Besides questionnaires (n = 65, 53%) and key indicators, usage logs are used to assess user loyalty (n = 15, 12%) (Figure 1).
Figure 1. Instruments that were used most frequently.
The dimension most frequently examined in all studies is the Use of Technology (n = 24, 20%). This examination is mainly carried out by usage logins (n = 15, 63%) and according to the theoretical construct of the “unified theory of acceptance and use of technology 2 (UTAUT2)” (n = 5, 21%). In addition, other important dimensions are usability (n = 22, 18%), quality of life (n = 20, 16%), and other economic measures (n = 22, 18%). The least used methods are those related exclusively to psychological well-being. Figure 2 shows the percentage of the 122 assessments conducted that apply to each dimension.
Figure 2. Dimensions that were used most frequently.
Among the 37 studies identified, just over half (n = 19, 51%) are mHealth systems, about a third are mHealth apps (n = 12, 32%), and 6 are applications for text messages. If the frequency of the implementation of the instruments is considered (Table 1), for mHealth systems (n = 63, 100%), both key indicators (n = 11, 17%) and usage logs (n = 10, 16%) are used almost equally. mHealth apps (n = 43, 100%) and mHealth text messaging (n = 16, 100%) are predominantly assessed by questionnaires.
Table 1. Type of intervention and the frequency of instruments.
3.2. Standardized and self-defined questionnaires
There are a total of 36 questionnaires among the evaluation tools. These include 28 standardized questionnaires with defined scores and eight individually defined questionnaires created by the app providers. Table 2 shows all questionnaires according to frequencies that occurred at least twice. Of the 28 standardized questionnaires, the validated generic EQ-5D questionnaire for measuring quality of life (n = 6, 21%) occurred most frequently.
Table 2. Standardized questionnaires that were used most frequently.
Overall, the contents of the questionnaires most frequently related to the usability (n = 10, 28%) and quality of life (n = 8, 22%). Among the ten questionnaires for usability, 8 were self-defined questionnaires, and only 2 were standardized questionnaires. It could be additionally determined that about 17% (n = 6) of the questionnaires are focused on disease-specific content, especially chronic and heart diseases. To assess medication adherence and psychological well-being while using the app, only questionnaires were used, such as the Morisky Medication Adherence Scale (MMAS-8) (19, 26–28, 47) or the Hospital Anxiety and Depression Scale (HADS) (23, 45, 47). Physical activities have been measured using the International Physical Activity Questionnaire (IPAQ) (16, 21, 31, 43, 45) and the Godin Leisure Time Physical Activity Questionnaire (47). In addition, the study by Beatty et al. (14) conducted a semi-structured interview to assess physical activity. Technology use is the least studied among all the questionnaires (n = 1, 3%).
3.3. Use of technology and usability
Five assessment tools were identified from the 37 studies to assess technology usage. These included usage logs, semi-structured interviews, the Mobile Application Rating Scale (MARS), and two model constructs. The model constructs included the UTAUT2 and the Technology Acceptance Model. The goal of the UTAUT2 is to analyze the behavioral intention to use a telemonitoring system. Various factors are considered and evaluated using quantitative and qualitative research methods (52). Similarly, the technology acceptance model describes the extent to which a person believes their work performance can be enhanced using the system (53).
To evaluate usability (n = 22), mainly self-defined questionnaires (n = 8, 36%) and semi-structured interviews (n = 5, 23%) have been used. In addition, three studies (14, 21, 30) used the standardized questionnaires “System Usability Scale” and one study (34) used the Perceived Health Web Site Usability Questionnaire.
3.4. Economic measurements and key indicators
Of the 37 studies, 12 (13, 16, 22, 25, 27, 29, 31, 35–38, 40) used key indicators to evaluate the app. Table 3 shows the frequency of instruments that used key indicators (n = 22) and the associated dimension.
Table 3. Number of instruments used, and the associated dimension.
Apart from the quality-adjusted life year (QALY) measure, the evaluation tools are almost exclusively for economic measures. The QALY is a measure that puts the lifetime gained by a measure in relation to the quality of life present in this time interval (54). The vast majority paid particular attention to hospital readmissions (n = 9, 41%) and mortality (n = 4, 18%). A specific cost-effectiveness analysis was performed in a total of 2 studies (19, 28).
4. Discussion
Based on the analysis, it has become clear that questionnaires are among the most frequently used evaluation tools for mHealth in CVD, which is the case for all application types (mHealth app, mHealth system, mHealth text messing). Quantitative methods mean less time and cost for researchers due to validated and meaningful data and are a popular method. Questionnaires are mainly used to improve usability and assess changes in quality of life. It is noticeable that predominantly standardized questionnaires were used with which a dimension can be evaluated specifically. However, self-defined questionnaires were primarily used to evaluate usability instead of the specific standardized questionnaires, System Usability and Perceived Health Web Site Usability Questionnaire. Heiney et al. (22) reasoned as follows (…) we were unable to identify an evaluation tool specifically for mHealth apps and this population. For example, the Systems Usability Scale was too broad to help us identify specific problems with the phone and the app. We asked closed questions that assessed potential problems with the phone and messages (i.e., readability) (…). The example shows that app providers do not yet use reliable, standardized evaluation tools to assess the usability of a mHealth application. Subjective questions, especially about usability and visual appeal, are challenging to standardize but are among the essential features that drive user engagement with apps (55). Open-ended feedback (33, 43, 48) or semi-structured interviews (14, 16, 19, 41, 49) are preferred to identify specific improvements to the app’s functionality.
Besides questionnaires, among the most widely used assessment tools are usage logs to evaluate user updates and adherence. Usage logs help track the patient’s interaction with the application. From this, patient motivation can be determined. Engagement and acceptance are essential to integrate mHealth interventions into care in a long-term and resource-efficient manner.
Comprehensive economic calculation bases occur only in 3 studies (16, 31, 40). The study by Cano Martín et al. (40) evaluates the economic impact of using a mobile app for the self-management of heart disease. To this end, a cost-effectiveness analysis was conducted. It is concluded that the app’s introduction could result in a 33% reduction in the cost of managing and treating the disease. From today’s perspective, savings like these are essential to cap limited resources and rising costs in healthcare systems. Another economic indicator for calculating the quality of life was examined in the study by Sankaran et al. (16). The improvement in quality of life by the mHealth application was measured using a QALY calculation. Calculations such as these provide the decisive impetus for health insurance companies to finance new forms of therapy.
Evaluation methods regarding data transfer between app providers and patients were not available. According to a survey (56), 45% of consumers expressed concerns related to the unintended use and sharing of personal health data. Therefore, to reduce consumer concerns, it is recommended to ensure a transparent presentation of results by evaluating a privacy system, e.g., “Privacy Management Platform.”
4.1. Limitations
Due to the scope of this study, we focused on individual-level instruments used to evaluate CVD mHealth applications. Individual medical measurements, such as various laboratory tests or vital sign measurements, were not considered due to the diversity and multiplicity of these indicators due to the heterogeneity of the applications included in the study. Nevertheless, in further studies, we would like to investigate the research need for methods to evaluate clinical endpoints for CVD apps. In addition, in a further study we plan to analyze the effects reported for each mHealth app among the studies with the aim of investigating to what extent the different types of evaluation do or do not lead to different ranges of effects.
4.2. Conclusion
This study aimed to analyze evaluation tools from 37 studies applied to CVD mHealth interventions. The resulting data shows that quantitative questionnaires, use logs, or key indicators are predominantly used. Regarding the questionnaires, currently, there are few standardized questionnaires to determine usability for mHealth applications. Here, a need for research arises for developing new or optimizing existing questionnaires. Existing theoretical constructs such as the UTAUT2 or the technology acceptance model, which can assess more in-depth individual use of the technology, are only used in a few studies. However, these should not be disregarded as such guideline mHealth interventions are used to optimize and improve for patients. A comprehensive framework that identifies and operationalizes the criteria for assessing mHealth applications can provide long-term policy recommendations on the impact of mHealth applications and is, therefore, essential for the further development of the healthcare system.
For this reason, economic metrics are also an essential consideration for evaluating mHealth applications. More profound calculations, such as cost-effectiveness analysis or QALY, should be included in an evaluation framework in addition to hospital KPIs (mortality, readmission, etc.) so that new forms of therapy, such as mHealth applications, can be reimbursed by health insurers in the future. mHealth apps have great potential to improve the quality and efficiency of services. To ensure quality, testing mHealth applications for their objective benefits is important.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
JK, WS, and FH contributed to the conception and design of the study and wrote sections of the manuscript. JK performed the data analysis and wrote the first draft of the manuscript. All authors contributed to the article and approved the submitted version.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1155433/full#supplementary-material
References
1. Santo, K, and Redfern, J. Digital Health Innovations to Improve Cardiovascular Disease Care. Curr. Atheroscler. Rep. (2020) 22:71. doi: 10.1007/s11883-020-00889-x
2. Holl, F, and Swoboda, W. Methods to measure the impact of mHealth applications: preliminary results of a scoping review. Stud Health Technol Inform. (2018) 251:285–8. doi: 10.3233/978-1-61499-880-8-285
3. Lagan, S, Sandler, L, and Torous, J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open. (2021) 11:e047001. doi: 10.1136/bmjopen-2020-047001
4. Kvedar, J, Coye, MJ, and Everett, W. Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Aff. (2014) 33:194–9. doi: 10.1377/hlthaff.2013.0992
5. Statistisches Bundesamt (Destatis). Gestorbene: Deutschland, Jahre, Todesursachen, Geschlecht (2021) Available at: https://www-genesis.destatis.de/genesis/online?sequenz=tabelleErgebnis&selectionname=23211-0002#abreadcrumb.
6. Nier, H. Herz-Kreislauf-Erkrankungen verursachen höchste Kosten (2017) Available at: https://de.statista.com/infografik/11301/herz-kreislauf-erkrankungen-verursachen-hoechste-kosten/.
7. Antonicelli, R, Testarmata, P, Spazzafumo, L, Gagliardi, C, Bilo, G, Valentini, M, et al. Impact of telemonitoring at home on the management of elderly patients with congestive heart failure. J Telemed Telecare. (2008) 14:300–5. doi: 10.1258/jtt.2008.071213
8. Dugas, M, Gao, G, and Agarwal, R. Unpacking mHealth interventions: a systematic review of behavior change techniques used in randomized controlled trials assessing mHealth effectiveness. Digit Health. (2020) 6:205520762090541. doi: 10.1177/2055207620905411
9. Rowland, SP, Fitzgerald, JE, Holme, T, Powell, J, and McGregor, A. What is the clinical value of mHealth for patients? NPJ Digit Med. (2020) 3:4. doi: 10.1038/s41746-019-0206-x
10. Sward, KA, Richard, M, Hickey, KT, and Moser, DK. Using mobile health technology for the management of chronic conditions. Nurse Pract. (2019) 44:34–9.
11. Yeh, RW, and Kramer, DB. Decision Tools to Improve Personalized Care in Cardiovascular Disease: Moving the Art of Medicine Toward Science. Circulation (2017) 135:1097–100. doi: 10.1161/CIRCULATIONAHA.116.024247
12. Holl, F, Kircher, J, Swoboda, WJ, and Schobel, J. Methods used to evaluate mHealth applications for cardiovascular disease: a quasi-systematic scoping review. Int J Environ Res Public Health. (2021) 18:12315. doi: 10.3390/ijerph182312315
13. Seto, E, Leonard, KJ, Cafazzo, JA, Barnsley, J, Masino, C, and Ross, HJ. Mobile phone-based telemonitoring for heart failure management: a randomized controlled trial. J Med Internet Res. (2012) 14:e31. doi: 10.2196/jmir.1909
14. Beatty, AL, Magnusson, SL, Fortney, JC, Sayre, GG, and Whooley, MA. VA FitHeart, a mobile app for cardiac rehabilitation: usability study. JMIR Hum Factors. (2018) 5:e3. doi: 10.2196/humanfactors.8017
15. Harzand, A, Witbrodt, B, Davis-Watts, ML, Alrohaibani, A, Goese, D, Wenger, NK, et al. Feasibility of a smartphone-enabled cardiac rehabilitation program in male veterans with previous clinical evidence of coronary heart disease. Am J Cardiol. (2018) 122:1471–6. doi: 10.1016/j.amjcard.2018.07.028
16. Sankaran, S, Dendale, P, and Coninx, K. Evaluating the impact of the HeartHab app on motivation, physical activity, quality of life, and risk factors of coronary artery disease patients: multidisciplinary crossover study. JMIR Mhealth Uhealth. (2019) 7:e10874. doi: 10.2196/10874
17. Guo, Y, Chen, Y, Lane, DA, Liu, L, Wang, Y, and Lip, GYH. Mobile health technology for atrial fibrillation management integrating decision support, education, and patient involvement: mAF app trial. Am J Med. (2017) 130:1388–1396.e6. e6. doi: 10.1016/j.amjmed.2017.07.003
18. Hickey, KT, Biviano, AB, Garan, H, Sciacca, RR, Riga, T, Warren, K, et al. Evaluating the utility of mHealth ECG heart monitoring for the detection and management of atrial fibrillation in clinical practice. J Atr Fibrillation. (2017) 9:1546. doi: 10.4022/jafib.1546
19. Lefler, LL, Rhoads, SJ, Harris, M, Funderburg, AE, Lubin, SA, Martel, ID, et al. Evaluating the use of mobile health technology in older adults with heart failure: mixed-methods study. JMIR Aging. (2018) 1:e12178. doi: 10.2196/12178
20. Goldstein, CM, Gathright, EC, Dolansky, MA, Gunstad, J, Sterns, A, Redle, JD, et al. Randomized controlled feasibility trial of two telemedicine medication reminder systems for older adults with heart failure. J Telemed Telecare. (2014) 20:293–9. doi: 10.1177/1357633X14541039
21. Maddison, R, Pfaeffli, L, Stewart, R, Kerr, A, Jiang, Y, Rawstorn, J, et al. The HEART mobile phone trial: the partial mediating effects of self-efficacy on physical activity among cardiac patients. Front Public Health. (2014) 2:56. doi: 10.3389/fpubh.2014.00056
22. Heiney, SP, Donevant, SB, Adams, SA, Parker, PD, Chen, H, and Levkoff, S. A smartphone app for self-management of heart failure in older African Americans: feasibility and usability study. JMIR Aging. (2020) 3:e17142. doi: 10.2196/17142
23. Freene, N, van Berlo, S, McManus, M, Mair, T, and Davey, R. A behavioral change smartphone APP and program (ToDo-CR) to decrease sedentary behavior in cardiac rehabilitation participants: prospective feasibility cohort study. JMIR Form Res. (2020) 4:e17359. doi: 10.2196/17359
24. Pandey, A, Krumme, A, Patel, T, and Choudhry, N. The impact of text messaging on medication adherence and exercise among postmyocardial infarction patients: randomized controlled pilot trial. JMIR Mhealth Uhealth. (2017) 5:e7144. doi: 10.2196/mhealth.7144
25. Park, C, Otobo, E, Ullman, J, Rogers, J, Fasihuddin, F, Garg, S, et al. Impact on readmission reduction among heart failure patients using digital health monitoring: feasibility and adoptability study. J Med Internet Res. (2019) 7:e13353. doi: 10.2196/13353
26. Santo, K, Singleton, A, Rogers, K, Thiagalingam, A, Chalmers, J, Chow, CK, et al. Medication reminder applications to improve adherence in coronary heart disease: a randomised clinical trial. Heart. (2019) 105:323–9. doi: 10.1136/heartjnl-2018-313479
27. Khonsari, S, Subramanian, P, Chinna, K, Latif, LA, Ling, LW, and Gholami, O. Effect of a reminder system using an automated short message service on medication adherence following acute coronary syndrome. Eur J Cardiovasc Nurs. (2015) 14:170–9. doi: 10.1177/1474515114521910
28. Park, LG, Howie-Esquivel, J, Chung, ML, and Dracup, K. A text messaging intervention to promote medication adherence for patients with coronary heart disease: a randomized controlled trial. Patient Educ Couns. (2014) 94:261–8. doi: 10.1016/j.pec.2013.10.027
29. Zan, S, Agboola, S, Moore, SA, Parks, KA, Kvedar, JC, and Jethwani, K. Patient engagement with a mobile web-based telemonitoring system for heart failure self-management: a pilot study. JMIR Mhealth Uhealth. (2015) 3:e3789. doi: 10.2196/mhealth.3789
30. Lunde, P, Nilsson, BB, Bergland, A, and Bye, A. Feasibility of a mobile phone app to promote adherence to a heart-healthy lifestyle: single-arm study. JMIR Form Res. (2019) 3:e12679. doi: 10.2196/12679
31. Maddison, R, Pfaeffli, L, Whittaker, R, Stewart, R, Kerr, A, Jiang, Y, et al. A mobile phone intervention increases physical activity in people with cardiovascular disease: results from the HEART randomized controlled trial. Eur J Prev Cardiol. (2015) 22:701–9. doi: 10.1177/2047487314535076
32. Lunde, P, Bye, A, Bergland, A, Grimsmo, J, Jarstad, E, and Nilsson, BB. Long-term follow-up with a smartphone application improves exercise capacity post cardiac rehabilitation: a randomized controlled trial. Eur J Prev Cardiol. (2020) 27:1782–92. doi: 10.1177/2047487320905717
33. Quilici, J, Fugon, L, Beguin, S, Morange, PE, Bonnet, J-L, Alessi, M-C, et al. Effect of motivational mobile phone short message service on aspirin adherence after coronary stenting for acute coronary syndrome. Int J Cardiol. (2013) 168:568–9. doi: 10.1016/j.ijcard.2013.01.252
34. Guo, X, Gu, X, Jiang, J, Li, H, Duan, R, Zhang, Y, et al. A hospital-community-family–based telehealth program for patients with chronic heart failure: single-arm, prospective feasibility study. JMIR Mhealth Uhealth. (2019) 7:e13229. doi: 10.2196/13229
35. Treskes, RW, van Winden, LAM, van Keulen, N, van der Velde, ET, Beeres, SLMA, Atsma, DE, et al. Effect of smartphone-enabled health monitoring devices vs regular follow-up on blood pressure control among patients after myocardial infarction: a randomized clinical trial. JAMA Netw Open. (2020) 3:e202165. doi: 10.1001/jamanetworkopen.2020.2165
36. Ware, P, Ross, HJ, Cafazzo, JA, Boodoo, C, Munnery, M, and Seto, E. Outcomes of a heart failure telemonitoring program implemented as the standard of care in an outpatient heart function clinic: pretest-posttest pragmatic study. J Med Internet Res. (2020) 22:e16538. doi: 10.2196/16538
37. Layton, AM, Whitworth, J, Peacock, J, Bartels, MN, Jellen, PA, and Thomashow, BM. Feasibility and acceptability of utilizing a smartphone based application to monitor outpatient discharge instruction compliance in cardiac disease patients around discharge from hospitalization. Int J Telemed Appl. (2014) 2014:1–10. doi: 10.1155/2014/415868
38. Chen, C, Li, X, Sun, L, Cao, S, Kang, Y, Hong, L, et al. Post-discharge short message service improves short-term clinical outcome and self-care behaviour in chronic heart failure. ESC Heart Fail. (2019) 6:164–73. doi: 10.1002/ehf2.12380
39. Park, LG, Howie-Esquivel, J, Whooley, MA, and Dracup, K. Psychosocial factors and medication adherence among patients with coronary heart disease: a text messaging intervention. Eur J Cardiovasc Nurs. (2015) 14:264–73. doi: 10.1177/1474515114537024
40. Cano Martín, JA, Martínez-Pérez, B, de la Torre-Díez, I, and López-Coronado, M. Economic impact assessment from the use of a mobile app for the self-management of heart diseases by patients with heart failure in a Spanish region. J Med Syst. (2014) 38:1–7. doi: 10.1007/s10916-014-0096-z
41. Woods, LS, Duff, J, Roehrer, E, Walker, K, and Cummings, E. Patients’ experiences of using a consumer mhealth app for self-management of heart failure: mixed-methods study. JMIR Hum Factors. (2019) 6:e13009. doi: 10.2196/13009
42. Ware, P, Dorai, M, Ross, HJ, Cafazzo, JA, Laporte, A, Boodoo, C, et al. Patient adherence to a Mobile phone–based heart failure Telemonitoring program: a longitudinal mixed-methods study. JMIR Mhealth Uhealth. (2019) 7:e13259. doi: 10.2196/13259
43. Chen, S, Gong, E, Kazi, DS, Gates, AB, Bai, R, Fu, H, et al. Using mobile health intervention to improve secondary prevention of coronary heart diseases in China: mixed-methods feasibility study. JMIR Mhealth Uhealth. (2018) 6:e7849. doi: 10.2196/mhealth.7849
44. Nabutovsky, I, Ashri, S, Nachshon, A, Tesler, R, Shapiro, Y, Wright, E, et al. Feasibility, safety, and effectiveness of a mobile application in cardiac rehabilitation. Isr Med Assoc J. (2020) 22:357–63.
45. Antypas, K, and Wangberg, SC. An internet-and mobile-based tailored intervention to enhance maintenance of physical activity after cardiac rehabilitation: short-term results of a randomized controlled trial. J Med Internet Res. (2014) 16:e77. doi: 10.2196/jmir.3132
46. Del Rosario, MB, Lovell, NH, Fildes, J, Holgate, K, Yu, J, Ferry, C, et al. Evaluation of an mHealth-based adjunct to outpatient cardiac rehabilitation. IEEE J Biomed Health Inform. (2017) 22:1938–48. doi: 10.1109/JBHI.2017.2782209
47. Pfaeffli Dale, L, Whittaker, R, Jiang, Y, Stewart, R, Rolleston, A, and Maddison, R. Text message and internet support for coronary heart disease self-management: results from the Text4Heart randomized controlled trial. J Med Internet Res. (2015) 17:e4944. doi: 10.2196/jmir.4944
48. Santo, K, Singleton, A, Chow, CK, and Redfern, J. Evaluating reach, acceptability, utility, and engagement with an app-based intervention to improve medication adherence in patients with coronary heart disease in the MedApp-CHD study: a mixed-methods evaluation. Med Sci. (2019) 7:68. doi: 10.3390/medsci7060068
49. Nundy, S, Razi, RR, Dick, JJ, Smith, B, Mayo, A, O’Connor, A, et al. A text messaging intervention to improve heart failure self-management after hospital discharge in a largely African-American population: before-after study. J Med Internet Res. (2013) 15:e53. doi: 10.2196/jmir.2317
50. Moher, D, Liberati, A, Tetzlaff, J, and Altman, DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. (2009) 6:e1000097. doi: 10.1371/journal.pmed.1000097
51. Boyatzis, R. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, CA, USA: SAGE (1998).
52. Venkatesh, V, Thong, JYL, and Xu, X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q. (2012) 36:157–78. doi: 10.2307/41410412
53. Davis, FD, (Ed) A technology acceptance model for empirically testing new end-user information systems: theory and results. Boston, MA, USA: Massachusetts Institute of Technology (1985).
54. Whitehead, SJ, and Ali, S. Health outcomes in economic evaluation: the QALY and utilities. Br Med Bull. (2010) 96:5–21. doi: 10.1093/bmb/ldq033
55. Alqahtani, F, and Orji, R. Insights from user reviews to improve mental health apps. Health Informatics J. (2020) 26:2042–66. doi: 10.1177/1460458219896492
Keywords: mHealth, mobile application, cardiovascular disease, evaluation, assessment, methods
Citation: Kircher J, Swoboda W and Holl F (2023) Examining standardized tools used for the evaluation of mobile health applications for cardiovascular disease. Front. Public Health. 11:1155433. doi: 10.3389/fpubh.2023.1155433
Edited by:
Santi Martini, Airlangga University, IndonesiaReviewed by:
Han Feng, Tulane University, United StatesParikipandla Sridevi, Indira Gandhi National Tribal University, India
Copyright © 2023 Kircher, Swoboda and Holl. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Felix Holl, felix.holl@hnu.de