Danah Hourani1
Simone Darling1,2,3
Eloise Cameron1
James Dromey1Louise Crossley1Sanji Kanagalingam4Frank Muscara1,5Amanda Gwee1,2,5Grace Gell1
Harriet Hiscock1,2,3,5,6
Vicki Anderson1,2,5*- 1Murdoch Children's Research Institute, Melbourne, VIC, Australia
- 2Department of Paediatrics, University of Melbourne, Parkville, VIC, Australia
- 3Centre for Community Child Health, Murdoch Children's Research Institute, Melbourne, VIC, Australia
- 4Curve Tomorrow, Melbourne, VIC, Australia
- 5The Royal Children's Hospital, Melbourne, VIC, Australia
- 6Health Services Research Unit, The Royal Children's Hospital, Melbourne, VIC, Australia
Embedding digital technologies in healthcare has the potential to streamline and personalize medical care. However, healthcare systems are often fragmented, and therefore achieving a truly integrated digital health program can be challenging. To promote a streamlined, evidence-based approach to implementing digital health solutions in a healthcare system, the Murdoch Children's Research Institute (MCRI) established the Digital Health Translation and Implementation Program (DHTI) bringing together clinicians, researchers and digital health experts. From the program commencement, frontline clinical innovators have collaborated with DHTI team members to develop and implement digital solutions to address pain-points in the healthcare system. Throughout this program, important lessons have been learnt relating to the development, evaluation and implementation of digital solutions in the healthcare system. This paper explores these lessons and makes recommendations for the successful implementation of digital health solutions in healthcare systems under five main categories: (1) design and usability, (2) stakeholder engagement and uptake, (3) project management and resourcing, (4) process and implementation, and (5) evaluation. Recommendations suggested here are designed to support future healthcare-based digital health programs to maximize the impact digital solutions can have on the healthcare system and patients.
Introduction
Evidence-based digital technology has the potential to revolutionize healthcare, decreasing access barriers such as long wait-times and remote locations and increasing efficiencies during time-limited consultations (1, 2). The value of technology in healthcare has been highlighted during the COVID-19 pandemic, when evidence-based digital solutions, such as telehealth mobile or web applications (“apps”) and online portals, ensured the continuity of healthcare delivery (3–6). Consequently, the community could easily access evidence-based information and guidance on a range of pediatric conditions, clinical care continued remotely and research studies could collect crucial data during the pandemic (7–9). Yet the practicalities of implementation and sustainability of digital solutions in healthcare remain complex to navigate. To address these challenges, academic medical centers developed various programs (10–16). The Murdoch Children's Research Institute (MCRI) established the Digital Health Translation and Implementation program (DHTI) - an integrated program of digital health research and clinical and community application. The DHTI program, which commenced in 2017, aims to identify healthcare-related pain-points at the Royal Children's Hospital (RCH), and explore, develop and validate digital solutions [e.g., both standalone smartphone or web applications or those embedded into the Electronic Medical Records (EMR)] to solve these problems.
This paper details the structure, activities and essential elements of DHTI and aims to provide healthcare systems with a framework for successful development, implementation and sustainability of evidence-based digital health initiatives. Three such initiatives, developed and implemented by the DHTI program, are used to demonstrate the lessons learnt through establishing the program.
DHTI Structure and Initial Consultation Framework
Team Structure
DHTI employs a multidisciplinary model including clinicians, researchers, and digital health experts and is supported by the Melbourne Children's Campus [“the Campus”: RCH, MCRI and University of Melbourne Department of Paediatrics (UMDP)], and regulatory, statistical and health economics experts. It leverages existing expertise within MCRI's digital health team and the in-house industry partnership with health technology company Curve Tomorrow (DHTI's working group), to identify, understand and address issues impacting delivery of clinical care (“pain-points”) identified by our clinical partner, The RCH (17). The program is managed by an executive team, with input from an International Advisory Group and a Program Advisory Committee (PAC), comprised of a multi-disciplinary group of Campus stakeholders who contribute to DHTI's identification of potential opportunities for implementation of digital health solutions.
An essential element of the DHTI program is the “clinical innovator” who has both clinical and digital health expertise and facilitates effective communication. They have protected time to identify and champion digital health opportunities, internally advocate for digital health solutions and work with the technical team to explore solution feasibility. Their input enables sustainable implementation within the healthcare system, embedding directly into clinical pathways, workflows and resources. In the late stages of development, clinical innovators facilitate knowledge transfer between clinical, research, innovation, implementation and evaluation working groups to assist with adopting digital solutions.
Campus Consultation
DHTI first consulted with Campus stakeholders across many RCH clinical programs about their pain-points. More than 40 interviews were conducted with RCH executives, heads of department, medical and allied health professionals, administration and support staff. Results were collated, and issues that met pre-determined selection criteria were retained. Criteria included: 1. The issue would be best solved using a digital solution, 2. Implementation was feasible given The RCH infrastructure, and 3. Solving the issue would align with the Campus strategic priorities (18). The selected issues were then considered by the DHTI executive and the PAC to determine the three most important issues, based on alignment with RCH pain-points, clinical, financial and operational impact, post-study implementation and sustainability.
MCRI Innovation Lifecycle Stages
For these initiatives, and subsequent digital solutions, DHTI developed a four step Innovation Lifecycle, to guide the development and implementation of solutions (Figure 1).
Figure 1. The MCRI innovation lifecycle stages.
Step I, Preparation and Exploration, explores the problem and potential solutions in detail. Step II, Development and Demonstration, evaluates feasibility, usability, efficacy and effectiveness, often through pilot studies. Step III, Implementation, aims to embed the digital solution into routine practice. Step IV focuses on sustainability and requires a long-term vision and “business as usual” model—key to successful delivery of DHTI's solutions. While these steps are designed as sequential (i.e., problem exploration is completed before development), it can be necessary to return to earlier steps if problems are detected (i.e., new features need to be added at implementation, necessitating returning to exploration).
DHTI Initiatives
DHTI focuses on developing “hospital-ready” digital solutions to address critical pain-points. Below we discuss three such initiatives that have successfully moved through the MCRI Lifecycle (Table 1). Each addresses a specific health problem and its corresponding digital solution, with clinical innovators working with clinicians, researchers, technology developers, behavior-change specialists and health care administration to take a product from step I (preparation and exploration) to step IV (sustainability), as per Figure 1. Lessons learnt throughout have been framed as recommendations in the following section.
Table 1. Description of DHTI initiatives.
Recommendations
While there were many learnings from the DHTI program, our working group determined that there were five main categories that captured the majority of recommendations for a general audience. Below we describe each of these recommendations within the context of the DHTI initiatives described in Table 1.
Design and Usability
Design thinking is an established, iterative, solutions-based approach for ill-defined problems, involving input from multiple stakeholders to generate innovative solutions (22). Useful methodologies include mapping the patient journey and developing personas (23). The DHTI model uses Curve Tomorrow's “Design Way” (24), which is based on several published design thinking methodologies (25–28)., which ensures end users' experience is at the forefront at all stages of the Innovation Lifecycle.
The basic design thinking steps are: empathize, define, ideate, prototype, test and implement (29). For empathize, define and ideate, DHTI encourages spending enough time “exploring” potential solutions to avoid committing to a specific solution too early. Often the solution from this process will not be digital, but can be tested using a simple, low-cost prototype such as a pencil and paper solution. For example, a prototype of The Drug Dosing Calculator was first developed using Shiny application, allowing for rapid modifications to the product's core content before any costly development or EMR integration work commenced.
Testing usability is a key component of the development and demonstration step (step II). However, there are few digital health-specific published standards for usability, outside of guidelines in the field of health informatics (30). For example, the usability of HeadCheck was evaluated in several stages (20). Firstly, semi-structured interviews were run with emergency department (ED) consultants, nurse practitioners, sports medicine physicians, neurosurgeons, neuropsychologists, and physiotherapists to collect feedback on the acceptability, feasibility and utility of the app. Secondly, a community survey was conducted, identifying that 83.3% (n = 15) of parents agreed that the app helped them decide when to seek medical help for their child. Lastly, a sample of parents attending the ED with their child after their suspected concussion was surveyed, with 85.7% (n = 6) of parents reporting the app had increased their awareness of the importance of concussion recovery, knowledge of safe recovery, and timing for safe return to school/normal activities. This information was crucial for refining HeadCheck's design and functionality and demonstrates the importance of evaluating usability across a range of settings.
Stakeholder Engagement and Uptake
Stakeholder engagement is increasingly encouraged to achieve sustainable research and clinical impact, yet target users are rarely included during conception or development phases (31). A principle critical to DHTI's success is involving stakeholders early in the Innovation Lifecycle, including end users, clinical champions and advocates who impact the decision-makers in the organization and/or the end users' behavior (32). In a pediatric context, involvement of stakeholders should be family-centered and include parents/guardians, teachers, siblings, family doctors and the children/adolescents themselves. The early involvement of stakeholders in problem and solution identification ensures a tailored solution design, facilitates engagement and creates a feeling of shared ownership (33). The ongoing involvement of stakeholders in the development and demonstration of the product ensures the product remains problem-focused, and the solution directly responds to target users' identified needs, maximizing the product's chance of successful long-term implementation and sustainability.
For community projects such as HeadCheck (20), industry partners [i.e., Australian Football League (AFL)] are critical to reach target audiences to allow research and clinical teams to canvass key insights as described in section Design and Usability. In this example, the AFL also advocated for the product, promoting it within their local and international networks and assisting with marketing and dissemination. The AFL were also aware of current and future industry-specific policies (e.g., concussion protocols), mandates and user pathways which are key in the Lifecycle of Innovation.
Project Management and Resourcing
For each DHTI initiative, a core group was established, led by a project manager and supported by project-specific advisors. The responsibilities of the core group included appointing a “product owner”/“decision maker,” development of a program logic, ensuring clear documentation (product specifications) and transparency of deliverables with key stakeholders, formal agreements (license, collaboration), a project risk matrix and establishing clear, agreed roles and responsibilities to smoothen project execution. Project managers reported to the DHTI executive monthly to facilitate monitoring of progress and budgets.
DHTI initiatives were supported by a comprehensive budget to evaluate feasibility of initiatives prior to development, and guide approaches to potential partners and funding bodies. It is key to account for implementation and sustainability costs, facilitating product longevity beyond the development and demonstration (step II). In our experience, many of the costs associated with the development, implementation and sustainability of digital health solutions are poorly understood by clinicians and researchers. These include costs for technical support, app/website hosting, software upgrades, and regulatory approvals.
Process and Implementation
Standardized processes and resources are important at both a program and project level. At the program level, DHTI uses a standardized intake and evaluation process for unearthing pain-points, and selecting projects to generate solutions. Since its inception, DHTI has assessed 17 digital projects as part of step I of the Innovation Lifecycle. Four (24%) products were excluded at phase 1 (Pre-screen), three (18%) products at phase 2 (Screen) and four (24%) at phase 3 (Due Diligence). Six products (35%) were selected to progress through to phases 4 and 5 (Recommendation, Approval and Commitment) before progressing to step II of the Innovation Lifecycle.
To assist with a standardized process at the project level, we developed the Digital Health Navigation System (Darling et al., unpublished manuscript), which is a “self-evaluation” tool for researchers and clinicians that consolidates resources and targeted information relevant to their product in an accessible, user-friendly format. It draws on published frameworks, models, standards, guidelines, and rating scales, relevant to implementing digital health technologies (e.g., developed by the World Health Organization) and emphasizes the importance of scientific rigor and evidence across all steps of the innovation process (34, 35). It aims to lift the standard of digital health, and creates accountability amongst all members involved in the innovation process.
While implementation comprises its own defined Step (Figure 1), it is critical to consider, from the outset of a project, that successful implementation of any health technology must occur within a complex multi-level system. Drawing on implementation science principles, the project team should consider not just the technology/product and the immediate user, but also the inner (immediate environment e.g., hospital department, school, GP clinic, home) and outer contexts (larger political, social and economic context) (36). Methods to explore and plan for this multi-level implementation include mapping existing organization systems and workflows involved in implementation (37), stakeholder engagement, usability measures, rigorous scientific evaluation (38–40) and planning for ongoing technical and clinical support post-launch, and budgeting appropriately. These aspects of planning can and should be revisited throughout the innovation lifecycle.
Evaluation
DHTI has incorporated evaluation across all its Innovation Lifecycle stages, including technical and clinical validation, usability, and cost factors. An important component here is ensuring products meet standards set by relevant regulatory and governance bodies (e.g., Therapeutic Goods Administration, Food and Drug Administration, organization ethics committees), in each jurisdiction of users. For example, for the Trial Symptom Tracker to be released internationally 6 months after its initial release in Australia, a written submission was required by the Apple app store including evidence that the product complies with ethics regulations and was approved in all countries it was to be released in (Australia, UK, The Netherlands, and Spain). In a fast moving, live project where continuity of data collection was paramount, this posed a significant budget and time challenge.
Accepted ethical principles across clinical and health research, including respect for persons, beneficence, justice, and autonomy should be considered from the preparation phase to sustainability phase by all parties (developers, clinicians, researchers). Although issues of privacy are often evaluated, there are potentially implications for data quality and management, changes in patient-physician relationships and equity of access of healthcare services (41, 42). New ethical challenges continue to arise, and it is important to share knowledge and resources about identifying those challenges and steps to address them including the Connected and Open Research Ethics (CORE) initiative (43), Transparency for Trust (T4T) principles (44), and ethical navigation aids (45). Ethics committees may struggle to stay updated with ethical challenges emerging in the digital health space, so it is incumbent upon the digital health team to proactively find and utilize such resources.
Scientific evaluation is a challenging process in the digital health context, where rapid changes in technology are routine, and are incompatible with gold-standard research evaluation methodologies such as randomized control trials (RCTs) with typically elaborate prerequisites for study design suitability (28). In 2016, more than 259,000 health-related apps were available in app stores for smartphone devices and current numbers are likely much higher (46). However, few have undergone any scientific evaluation to determine effects on health outcomes, rather than use or user satisfaction, limiting their potential to impact clinical outcomes (47–49).
Optimally, evaluation is iterative and broad, based on needs and feedback, enabling the team to pivot to achieve the best solution. Digital health researchers need to identify novel, flexible approaches to evaluation and validation, and embrace real-world trials and evidence (50). DHTI's multicenter drug dosing calculator study, as mentioned in Table 1, exemplifies how to address this challenge through clearly defined research evaluation as an alternative methodology to an RCT design in digital health evaluation.
Discussion
This article details the DHTI structure and activities, and proposed a framework to successfully develop, implement and sustain evidence-based digital health initiatives.
The lessons learnt from the Melbourne Children's Campus can be summarized as follows:
1. Maintain a problem-centered approach and do not neglect the initial exploration stage
2. Ensure true buy-in and representation from all relevant stakeholders by establishing a cross-disciplinary team of relevant consultants with expertise in software development, knowledge translation, behavior change, statistics, health economics, regulatory processes and healthcare
3. Budget accurately and consider “hidden” costs such as maintenance and support fees
4. Standardize processes and resources for a varied audience (clinicians, researchers) to develop, evaluate and implement solutions
5. Develop an evaluation framework that suits your local context, is flexible and applicable to a digital health content. Conduct thorough evaluation at every step of the process.
These important lessons should be interpreted within the context of several limitations. Firstly, while our team is confident of the utility of the MCRI Innovation Lifecycle and the DHNS, given they are based on published literature, multi-stakeholder expert input and extensive experience, neither resources have been empirically validated. Our team plans to evaluate the MCRI Innovation Lifecycle framework against other published innovation processes for usability and utility. Further, we plan to evaluate the DHNS using key success metrics e.g., reduction in adverse events, number of products successfully implemented in standard care, increased life expectancy of digital health products. Secondly, as DHTI is in its infancy, the guidance we provide around long-term sustainability (step IV) is limited. As the program matures, we will gather more learnings around sustainability of digital health solutions. Lastly, while our program emphasizes the inclusion of feedback from key stakeholders, we have not yet been able to incorporate feedback from young children and adolescents. This is a major future goal of the program and will be crucial to implementing successful products in a pediatric setting.
Concluding Remarks
The novelty of digital health and its potential to rapidly impact and transform healthcare can raise concerns about credibility, reliability and success. A commitment by digital health programs such as DHTI to consistently, and rigorously evaluate at every step and incorporate feedback, will be essential to make stakeholders trusting and open to digital solutions.
DHTI successfully developed and implemented digital health solutions to solve real-world pain-points in the healthcare system, with important learnings captured throughout the process. While the DHTI program is by no means a perfect solution and will need to evolve with time, the key concepts described here are considered the most essential by the DHTI team for any digital health project. By prioritizing scientific rigor, ensuring standardized processes throughout the innovation lifecycle, and drawing on input from multi-stakeholders, programs such as DHTI play a crucial role in advocating for digital technologies' potential to transform how we deliver healthcare, optimizing care for our patients and broader community through rapid innovation and implementation.
As the first digital native generation approaches adulthood, the integration of digital technology into the lives of patients and the broader community will only increase. Healthcare systems need to embrace this digital revolution and ensure that the development of new processes and systems exist and the cultural shift has occurred to maximize readiness to adopt these solutions (51).
Data Availability Statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.
Author Contributions
SD, DH, VA, and HH contributed the conception and brainstorming of the article. DH organized the structure of the article and wrote the manuscript. All authors contributed to the article and approved the submitted version.
Funding
This work was supported by The Royal Children's Hospital Foundation (grant number 2017-877) and the Victorian Government's Operational Infrastructure Support Program supporting research at the Murdoch Children's Research Institute. HH was supported by NHMRC Practitioner Fellowship Award 1136222.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The authors would like to thank the Digital Health Translation and Implementation and Curve Tomorrow teams for their contribution to this work.
References
1. Laurenza E, Quintano M, Schiavone F, and Vrontis D. The effect of digital technologies adoption in healthcare industry: a case based analysis. Buss Process Manag J. (2018) 24:1124–44. doi: 10.1108/BPMJ-04-2017-0084
2. Imison C, Castle-Clarke S, Watson R, and Edwards N. Delivering the Benefits of Digital Health Care. London: Nuffield Trust (2016). Available online at: https://www.nuffieldtrust.org.uk/research/delivering-the-benefits-of-digital-health-care (accessed January 28, 2021).
3. Jnr BA. Use of telemedicine and virtual care for remote treatment in response to COVID-19 pandemic. J Med Syst. (2020) 44:1–9. doi: 10.1007/s10916-020-01596-5
4. Kapoor A, Guha S, Das MK, Goswami KC, and Yadav R. Digital healthcare: the only solution for better healthcare during COVID-19 pandemic?. Indian Heart J. (2020) 72:61–4. doi: 10.1016/j.ihj.2020.04.001
5. Hantrais L, Allin P, Kritikos M, Sogomonjan M, Anand PB, Livingstone S, et al. Covid-19 and the digital revolution. Contemp Soc Sci. (2020) 30:1–5. doi: 10.1080/21582041.2020.1833234
6. Haggerty E. Healthcare and digital transformation. Netw Sec. (2017) 8:7–11. doi: 10.1016/S1353-4858(17)30081-8
7. Badawy SM, and Radovic A. Digital approaches for remote pediatric healthcare delivery during the coronavirus (Covid-19) pandemic: existing evidence and a call for further research. JMIR Pediatr Parent. (2020) 1:e20049. doi: 10.2196/20049
8. Armoiry X, Sturt J, Phelps EE, Walker CL, Taggart F, Sutcliffe P, et al. Digital clinical communication for families and caregivers of children or young people with short-or long-term conditions: rapid review. J Med Int Res. (2018) 20:e5. doi: 10.2196/jmir.7999
9. Chambers R, and Marc S. Making technology-enabled health care work in general practice. Br J Gen Pract. (2018) 68:108–9. doi: 10.3399/bjgp18X694877
10. Bul K, Holliday N, Magee P, and Wark P. From development to exploitation of digital health solutions: lessons learnt through multidisciplinary research and consultancy. J Enabling Technol. (2020) 14:273–9. doi: 10.1108/JET-09-2020-0035
11. Ford KL, Moore SL, Zhou S, Gore MO, Portz J, Zhang X, et al. Advancing evidence-based digital health through an innovative research environment: an academic-industry collaboration case report. Mhealth. (2019) 5:37. doi: 10.21037/mhealth.2019.08.08
12. Mann DM, Chokshi SK, Lebwohl R, Mainiero M, Dinh-Le C, Driscoll K, et al. Building digital innovation capacity at a large academic medical center. NPJ Dig Med. (2019) 2:1–6. doi: 10.1038/s41746-019-0088-y
13. Tseng J, Samagh S, Fraser D, and Landman AB. Catalyzing healthcare transformation with digital health: performance indicators and lessons learned from a Digital Health Innovation Group. InHealthcare. Elsevier. (2018) 6:150–5. doi: 10.1016/j.hjdsi.2017.09.003
14. Chokshi SK, and Mann DM. Innovating from within: a process model for user-centered digital development in academic medical centers. JMIR Hum Factors. (2018) 5:e11048. doi: 10.2196/11048
15. DePasse JW, Chen CE, Sawyer A, Jethwani K, and Sim I. Academic medical centers as digital health catalysts. InHealthcare. (2014) 2:173–6. doi: 10.1016/j.hjdsi.2014.05.006
16. Ostrovsky A, and Barnett M. Accelerating change: fostering innovation in healthcare delivery at academic medical centers. InHealthcare. (2014) 2:9–13. doi: 10.1016/j.hjdsi.2013.12.001
17. Macleod P Dromey J. Digital health at the Murdoch children's research institute-a new model for the convergence of health and technology. Aust Biotechnol. (2015) 1:14. Available online at: https://search.informit.org/doi/10.3316/informit.091193891753362
18. The Royal Children's Hospital. Our Strategic Plan: Our Priorities (2019). Available online at: https://www.rch.org.au/strategic-plan/plan/our-priorities/ (accessed January 27, 2021).
19. Davis G, Thurairatnam S, Feleggakis P, Anderson V, Bressan S, and Bable FE. HeadCheck: a concussion app. J Paediatr Child Health. (2015) 51:830–1. doi: 10.1111/jpc.12879
20. Clarke C, Anderson V, Babl FE, Rausa VC, Davis GA, Barnett P, et al. Child concussion recognition and recovery: a community delivered, evidenced-based solution. Ann Transl Med. (2020) 8:595. doi: 10.21037/atm.2020.03.50
21. Gwee A, Cranswick N, McMullan B, Perkins E, Bolisetty S, Gardiner K, et al. Continuous versus intermittent vancomycin infusions in infants: a randomized controlled trial. Pediatrics. (2019) 143:e20182179. doi: 10.1542/peds.2018-2179
23. Holderfield G. Generating How Might We (HMW) Questions from Insights - Using Design Principles to Innovate and Find Opportunities. Coursera (2017). Available online at: https://www.coursera.org/learn/leadership-design-innovation/lecture/JMRYd/generatinghow-might-we-hmw-questions-from-insights (accessed December 18, 2020).
24. Curve Tomorrow. Curve's Design Way. Available online at: https://www.curvetomorrow.com.au/ (accessed November 18, 2020).
25. Stanford D. School. Available online at: https://dschool.stanford.edu/ (accessed January 29, 2021).
26. Google Sprint. The Design Sprint. Available online at: https://www.gv.com/sprint/ (accessed January 29, 2021).
27. IDEO. What is Design Thinking? Available online at: https://www.ideou.com/blogs/inspiration/what-is-design-thinking?_pos=4&_sid=edf5ac29e&_ss=r (accessed January 29, 2021).
28. Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med. (2016) 51:843–51. doi: 10.1016/j.amepre.2016.06.008
29. Wolniak R. The Design Thinking Method and Its Stages. Zabrze: Systemy Wspomagania Inzynierii Produkcji 6 (2017).
30. Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, et al. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc. (2013) 20:e2–8. doi: 10.1136/amiajnl-2012-001458
31. Fougerouse PA, Yasini M, Marchand G, and Aalami OO. A cross-sectional study of prominent US mobile health applications: evaluating the current landscape. AMIA Annu Symp Proc. (2017) 2017:715–23.
32. Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, et al. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J Gen Int Med. (2016) 31:13–21. doi: 10.1007/s11606-015-3450-z
33. Papoutsi C, Wherton J, Shaw S, Morrison C, and Greenhalgh T. Putting the social back into sociotechnical: case studies of co-design in digital health. J Am Med Inform Assoc. (2020) 28:284–93. doi: 10.1093/jamia/ocaa197
34. National Institute for Health Care and Care Excellence. Evidence Standards Framework for Digital Health Technologies. London: NICE (2019).
35. World Health Organization. Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment. Geneva: WHO (2016).
36. Lyon AR, and Bruns EJ. From evidence to impact: joining our best school mental health practices with our best implementation strategies. School Mental Health. (2019) 11:106–14. doi: 10.1007/s12310-018-09306-w
37. Gupta A, Thorpe C, Bhattacharyya O, and Zwarenstein M. Promoting development and uptake of health innovations: the nose to tail tool. F1000Research. (2016) 5:361. doi: 10.12688/f1000research.8145.1
38. Sligo J, Gauld R, Roberts V, and Villa L. A literature review for large-scale health information system project planning, implementation and evaluation. Int J Med Inform. (2017) 97:86–97. doi: 10.1016/j.ijmedinf.2016.09.007
39. Jones SS, Koppel R, Ridgely MS, Palen TE, Wu S, and Harrison MI. Guide to Reducing Unintended Consequences of Electronic Health Records. Rockville, MD: Agency for Healthcare Research and Quality (2011).
40. Hoffman S. Employing e-health: the impact of electronic health records on the workplace. Kan J Law Pub Pol'y. (2009) 19:409. Available online at: https://heinonline.org/HOL/LandingPage?handle=hein.journals/kjpp19&div=27&id=&page=
41. Caiani E. Ethics of digital health tools, European Society of Cardiology (2020). 18 p. Available online at: https://www.escardio.org/Journals/E-Journal-of-Cardiology-Practice/Volume-18/ethics-of-digital-health-tools (accessed April 18, 2021).
42. Nebeker C, Bartlett Ellis RJ, and Torous J. Development of a decision-making checklist tool to support technology selection in digital health research. Transl Behav Med. (2020) 10:1004–15. doi: 10.1093/tbm/ibz074
43. Nebeker C, Torous J, and Ellis RJB. Building the case for actionable ethics in digital health research supported by artificial intelligence. BMC Med. (2019) 17:1–7. doi: 10.1186/s12916-019-1377-7
44. Wykes T, Lipshitz J, and Schueller SM. Towards the design of ethical standards related to digital mental health and all its applications. Curr Treat Opt Psychiatry. (2019) 6:232–42. doi: 10.1007/s40501-019-00180-0
45. Gasser U, Ienca M, Scheibner J, Sleigh J, and Vayena E. Digital tools against COVID-19: taxonomy, ethical challenges, and navigation aid. Lancet Digital Health. (2020) 2:E425–34. doi: 10.1016/S2589-7500(20)30137-0
46. Research2guidance. The mHealth Apps Markter Is Getting Crowded. Available online at: https://research2guidance.com/mhealth-app-market-getting-crowded-259000-mhealth-apps-now/ (accessed January 29, 2021).
47. McMillan B, Hickey E, Mitchell C, and Patel M. The need for quality assurance of health apps. BMJ. (2015) 4:351. doi: 10.1136/bmj.h5915
48. Boulos MN, Brewer AC, Karimkhani C, Buller DB, and Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform. (2014) 5:229. doi: 10.5210/ojphi.v5i3.4814
49. Yasini M, and Marchand G. Toward a use case based classification of mobile health applications. InMIE. (2015) 210:175–9.
50. Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, and Cohen AB. Digital health: a path to validation. NPJ Dig Med. (2019) 2:1–9. doi: 10.1038/s41746-019-0111-3
Keywords: digital health (eHealth), evidence-based & research methodology, pediatrics - children, health system - organization and administration, mobile application (app)
Citation: Hourani D, Darling S, Cameron E, Dromey J, Crossley L, Kanagalingam S, Muscara F, Gwee A, Gell G, Hiscock H and Anderson V (2021) What Makes for a Successful Digital Health Integrated Program of Work? Lessons Learnt and Recommendations From the Melbourne Children's Campus. Front. Digit. Health 3:661708. doi: 10.3389/fdgth.2021.661708
Received: 31 January 2021; Accepted: 26 April 2021;
Published: 24 May 2021.
Edited by:
Wendy Chapman, The University of Melbourne, AustraliaReviewed by:
Yunan Chen, University of California, Irvine, United StatesLin Song, University of California, San Francisco, United States
Copyright © 2021 Hourani, Darling, Cameron, Dromey, Crossley, Kanagalingam, Muscara, Gwee, Gell, Hiscock and Anderson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Vicki Anderson, vicki.anderson@rch.org.au