Development and delivery cost of digital health technologies for mental health: Application to the Narrative Experiences Online Intervention

Background: The increasing development and use of digital health interventions requires good quality costing information to inform development and commissioning choices about resource allocation decisions. The Narrative Experiences Online (NEON) Intervention is a web-application that delivers recorded mental health recovery narratives to its users. Two randomized controlled trials are testing the NEON Intervention in people with experience of psychosis (NEON) and people experiencing non-psychosis mental health problems (NEON-O).

Aim: This study describes and estimates the cost components and total cost of developing and delivering the NEON Intervention.

Materials and methods: Total costs for the NEON Trial (739 participants) and NEON-O Trial (1,024 participants) were estimated by: identifying resource use categories involved in intervention development and delivery; accurate measurement or estimation of resource use; and a valuation of resource use to generate overall costs, using relevant unit costs. Resource use categories were identified through consultation with literature, costing reporting standards and iterative consultation with health researchers involved in NEON Intervention development and delivery. Sensitivity analysis was used to test assumptions made.

Results: The total cost of developing the NEON Intervention was £182,851. The largest cost components were software development (27%); Lived Experience Advisory Panel workshops (23%); coding the narratives (9%); and researchers' time to source narratives (9%). The total cost of NEON Intervention delivery during the NEON Trial was £118,663 (£349 per NEON Intervention user). In the NEON-O Trial, the total delivery cost of the NEON Intervention was £123,444 (£241 per NEON Intervention user). The largest cost components include updating the narrative collection (50%); advertising (19%); administration (14%); and software maintenance (11%). Uncertainty in the cost of administration had the largest effect on delivery cost estimates.

Conclusion: Our work shows that developing and delivering a digital health intervention requires expertise and time commitment from a range of personnel. Teams developing digital narrative interventions need to allocate substantial resources to curating narrative collections.

Implications for practice: This study identifies the development and delivery resource use categories of a digital health intervention to promote the consistent reporting of costs and informs future decision-making about the costs of delivering the NEON Intervention at scale.

Trial registration: NEON Trial: ISRCTN11152837, registered 13 August 2018, http://www.isrctn.com/ISRCTN11152837. NEON-O Trial: ISRCTN63197153, registered 9 January 2020, http://www.isrctn.com/ISRCTN63197153.

Introduction

In recent years, there has been a rapid increase in digital health technologies (DHTs) to deliver mental health interventions remotely, either to replace or supplement face-to-face healthcare, with that increase accelerating during the COVID-19 pandemic (1, 2).

Digital health technologies range from predefined functions such as remote diagnosis or disease monitoring, to more complex functions around supporting behavior change through interactive and personalized interactions. A sub-type of DHTs are digital health interventions (DHIs), which provide users with remote access to treatment through text messaging services, smartphone apps, and web-based resources. Recent reviews of the growing number of trials have shown promising but varied effectiveness for DHIs in a wide range of mental health conditions, including but not limited to depression (3, 4), eating disorders (5), post-traumatic stress disorder (6), and schizophrenia (7).

Investment in mental health DHIs is increasing, partly to improve access to care in overstretched health systems and partly due to emerging patient and user preference for DHI delivery of mental health care (4). This means that decisions are being made about effectiveness and affordability of individual DHIs. Economic evaluations inform healthcare resource allocation decisions and treatment recommendations by comparing the costs and health benefits of alternative ways to treat patients (8). Demonstrating cost-effectiveness is an important factor for delivering DHIs into healthcare systems across Europe (9). Economic evaluations of mental health DHIs have been conducted for a range of conditions such as anxiety (10, 11), depression (10–12), eating disorders (10, 13), and substance abuse disorders (10, 14). The quality of economic evaluations has been suggested to be very variable including heterogeneous reporting of costs, outcomes and comparators (15). Such inconsistencies may be a barrier to the adoption of effective DHIs for mental health within healthcare systems or may lead policymakers to invest in services that are not cost-effective (16–18).

There is a lack of standardization in how costs are reported and described for economic evaluations of DHIs (11, 15). The National Institute of Health and Care Excellence's (NICE) evidence standards framework for DHTs recommends reporting all costs associated with the intervention and costs relevant to a health and social care decision maker (19). The framework focuses on delivery (implementation) costs, as these are relevant to health and social care providers looking to commission a service, and includes initial investment costs such as training, as well as operation, and maintenance costs (19). Apart from the initial fixed set-up costs, including infrastructure and training costs, evolution of the DHI over time (interface design, software updates, and content updates) requires more flexible data collection tools that can keep pace with these changes. For example, the marginal cost is the additional cost incurred when one more person uses the product. DHIs tend to have a low marginal cost (one more user accessing an app tends to a zero marginal cost) up to a certain threshold, beyond which there is a large increase due to the need to re-engineer components of technology (such as centralized servers maintaining user account records and delivering web-based content) to provide additional capacity. Other challenges for DHI costing include having to respond to rapidly changing prices (such as exchange rate fluctuations) and short life-cycles of technology, short depreciation periods, and attribution of cost from a shared resource such as a wireless network (20).

Development costs are not typically considered in economic evaluations as, by definition, they have already been incurred prior to delivery. Historically, these development costs are sunk costs, and may include cross-subsidization of product development where there is a failure to reach product launch. Development costs can be included in a market price, which will fall as the market share, or the number of users increases. A recent systematic review reported development costs were only reported in four out of 24 economic evaluations of internet-based interventions for anxiety and depression (11). There are no guidelines for estimating the development cost of DHIs, resulting in non-standardized approaches and a lack of comparability across studies. However, development costs are relevant to publicly funded research groups, research funders, university enterprise offices and private companies and by informing decisions to develop DHIs or scale an existing technology. DHI development incurs significant research and development costs prior to launch, and the resources consumed during development can be very different from pharmaceuticals and physical medical devices. DHIs have other unique characteristics that can affect development processes and thus costs, including faster evolution, active and more dynamic user input (21). DHI development costs are usually incurred for content and software design, website and graphic design, digital platform development and regulatory approval processes. A recent study focusing on the development of a mental health DHI proposed a costing framework for development costs and we have applied this framework to our development costs methods (22).

The aim of this study is to describe, estimate, and present the associated cost components and total cost of developing and delivering the Narrative Experiences Online (NEON) Intervention, using current recommendations for costing DHIs (11, 19, 21, 22).

Digital health intervention under investigation: NEON Intervention

The NEON Intervention is a web-based application that delivers recorded recovery narratives to its users. A systematic review (23) and qualitative validation study (24) defined mental health recovery narratives as first-person lived experience accounts of recovery from mental health problems, which include elements of adversity or struggle and of self-defined strengths, successes or survival. The impact of recovery narratives was then investigated in a systematic review (25), qualitative interviews (26), and experimental studies (27, 28). Approaches to curation of recorded recovery narrative collections were developed through systematic review (29), stakeholder consultation (30) and best practice guidelines development (31). These studies, together with related work on post-traumatic growth (32, 33), non-service user perspectives (34), institutional injustice (35), and clinician perspectives on use of narratives in practice (36), provided the theory base.

The NEON Intervention was then developed (37). The NEON Intervention allows recipients to engage with recovery narratives by watching, reading, or listening to narrator-led stories on the website. Access to these narratives is provided through different avenues: a hybrid recommender system using collaborative and content-based filtering to recommend appropriate stories; self-selected stories from the entire collection of narratives (referred to as the NEON Collection); randomly selected stories; recommendations sent to users in emails, to serve as a mechanism for engaging people with the intervention; and re-requested narratives that have been previously seen. The recommender system uses feedback data from stories received by participants, characteristics of the participants, and characteristics of each recorded recovery narrative assessed using the Inventory of Characteristics of Recovery Stories (INCRESE) measure (38), to match participants to narratives intended to be of benefit. The website was engineered to work on personal computers, mobile devices, or communal computers such as in a public library to enable participation by people experiencing digital exclusion (39).

An economic evaluation is being conducted as part of two definitive randomized controlled trials (RCTs) to evaluate the cost-effectiveness of offering the NEON Intervention to individuals with experience of psychosis (NEON Trial, n = 739) and people experiencing non-psychosis mental health problems (NEON-O Trial, n = 1,024) (40, 41).

This study reports the estimation of development and delivery costs for the NEON Intervention, which is used in both trials.

Materials and methods

This study was conducted as part of the Narrative Experiences Online (NEON) Programme (researchintorecovery.com/neon), which is investigating whether receiving recorded mental health recovery narratives improves the quality of life in people who experience mental health issues.

Design of costing study

Estimated total cost and individual cost components for the development and delivery of the NEON Intervention are reported. The development costs included those incurred from the perspective of the research body who funded the development and testing of the NEON Intervention. The delivery costs included those incurred from the perspective of the health and social care provider (National Health Service, NHS England) (19). The length of time over which delivery costs were collected was from the beginning of delivery (9th March 2020) to the end of the trial periods (13th May 2022 for NEON Trial, 23rd June 2022 for NEON-O Trial). The NEON Intervention was identical for both NEON and NEON-O trials, since both trials provided participants with access to the same narrative collection. A separate cost of delivery of the NEON Intervention is presented for both trials. Downstream costs (those incurred as a result of using the intervention from the perspective of the NHS) will be reported in a subsequent full economic evaluation.

The development costs begin from the start of the current research program funding, comprising software and intervention development, through feasibility testing and up to the starting point of the definitive trial. Early conceptual development work, and research-related tasks including the randomized controlled trials which contribute to a large part of the program grant, are excluded in the development cost estimates. The duration of the development of the NEON Intervention was considered over the period January 2017 to January 2021.

The total cost for the development stage of the NEON Intervention was constructed with reference to a development costing checklist for a digital program for training community health workers to deliver treatment for depression (22). The costing strategy consisted of: identifying resource use categories involved in the development and delivery of the intervention; accurate measurement or estimation of resource use; and a valuation of resource use to generate overall costs (42). Identifying resource use categories was carried out through consultation with published literature relevant to DHIs, costing reporting standards and expert consultation with team members involved in intervention development and delivery.

The product of resource use and unit costs generated total cost estimates, and this is referred to as the base-case analysis. In this study, the resource use data obtained for development and delivery was obtained for the whole user cohort (top-down costs), rather than data relevant to a specific user (bottom-up costs). Therefore, the approach taken was to generate total development and delivery costs and then apportion to individual users as a “mean cost per user.” The number of users was defined as the number of people randomized to the intervention group (370 in the NEON Trial and 512 in the NEON-O Trial).

In the base-case analysis, costs for human resources were obtained by multiplying the personnel's midpoint hourly salary plus on-costs (pension contributions and payroll taxes) by the proportion of hours spent on the task. For external experts and consultants their costs were recorded as invoices to the NEON study. In some cases where records were not kept, the proportion of hours spent on a task was estimated through expert consultation to produce the maximum and minimum plausible duration; with the midpoint (average) selected during the base-case analysis. For information technologies used in the development and delivery of the NEON Intervention, financial records were used to calculate the cost of the components purchased. In one case, an estimate was derived from the NEON budget proposal.

Deriving estimates of certain cost components required some assumptions to be made with uncertainty further associated with the true values of several components. To assess the impact of our assumptions and parameter uncertainty, all input parameters were adjusted to their extreme values individually in a one-way sensitivity analysis. Tornado diagrams (43) were used to illustrate which input parameters had the most impact on cost estimates. Structural uncertainty resulting from the assumptions made were examined through scenario analysis.

This costing study was developed and reported in accordance to standard validation and reporting criteria (44). A team member not involved in the analysis used the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 checklist to ensure that relevant items were reported completely (see Appendix 1). Face validity of cost categories and costing methods was ascertained through continuous feedback from clinical and patient experts. All costs are presented in UK Sterling (£) for the costing period 2020–2021.

Costing the NEON Intervention

We identified several resource use categories in intervention development and delivery, summarized in Table 1. This section provides detail on how resource use and unit costs were obtained for each category.

TABLE 1

Table 1. Summary of NEON Intervention development and delivery resource use categories.

NEON Intervention development resource use

Four components that incurred resources were needed to develop the NEON Intervention:

• building a collection of recovery narratives (the NEON Collection) that would be used in the intervention (consisting of sourcing narratives, securing ethical approval to use those narratives, liaising with collection organizers to source narratives from existing collections, Lived Experience Advisory Panel (LEAP) workshops to develop recommendations on (1) the ethical issues around narrative curation and (2) the initial curation procedures, Collection Steering Group (CSG) meetings to make decisions on (1) the inclusion of individual narratives and (2) refinement of the curation procedures, training researchers to use the INCRESE tool, coding narratives using INCRESE);

• developing the web-application as a platform to deliver the NEON Intervention [consisting of reporting a database specification to support the development of the web-application, developing source code for the web-application (software development), conceptualizing and developing a recommender system that matches users with the most appropriate narratives, integrating the recommender system into the web-application codebase, and designing the intervention to ensure it was appealing to both operate aesthetically (graphic design) and practically (interaction design)];

• additional communications necessary to develop the intervention [consisting of NEON study team task meetings for general discussions, International Advisory Board (IAB) meetings to advise on safety and ethical concerns, and Intervention Development Group (IDG) meetings to provide (feedback on the intervention)];

• testing the intervention (consisting of testing the intervention's performance on the web-application including the collection of outcome data, and a feasibility study evaluating a prototype of the intervention in a small sample of mental health service users (baseline: n = 25; follow-up: n = 22) with experience of mental health problems);

Resource use during the development of the NEON Intervention was measured through examinations of records and/or derived estimates with assistance from expert consultation. Tables 2–5 summarize resource use and unit cost categories and sources for the four components of NEON intervention development.

TABLE 2

Table 2. Development resource use and unit cost input parameters: NEON Collection curation.

TABLE 3

Table 3. Development resource use and unit cost input parameters: Web-application development.

TABLE 4

Table 4. Development resource use and unit cost input parameters: Additional communication.

TABLE 5

Table 5. Development resource use and unit cost input parameters: Testing the intervention.

An essential component of the development of the NEON Intervention was building the NEON Collection (37). The recovery stories used were sourced based on the objective of maximizing the diversity of the types of stories within the collection based on different diversity domains (such as narrator ethnicity, sexuality, gender identity, neurodiversity, etc.). There were two routes in which narratives were sourced for the collection: individual donations (~7% of the collection) and existing collections (~93% of the collection). Both routes required work to secure permission to use the narratives within the NEON Collection. For donated narratives, this involved liaising directly with the narrator to secure permission for the re-use of the story. For narratives sourced from existing collections, the curator either had prior permission for the narratives to be redistributed or they approached the narrators for whom they did not already have the appropriate permission for the re-use of the narratives. The eligibility for a narrative to be adopted within the collection was assessed based on the inclusion/exclusion criteria identified through NEON task meetings and Lived Experience Advisory Panel (LEAP) workshops with any ethical uncertainties resolved during Collection Steering Group (CSG) meetings.

The NEON study LEAP workshops were chaired 13 times with attendance from 10 LEAP members, a LEAP meeting chair, and members of the NEON team. It was estimated that 60% of these workshops were necessary for the development of NEON as opposed to other research-related activities by examining the workshop agendas. The LEAP members had personal experiences of mental health problems and advised on the ethical principles of curating narratives, categorizing narrative content warnings, general issues raised by the research team, and the types of narratives to be included/excluded from the NEON Collection (28). As with all in-person meetings/workshops run by the NEON study all participants were paid for their attendance. Additionally, before the COVID-19 pandemic, travel expenses and hospitality (including venues and refreshments) were covered as expenses and recorded through invoices.

A CSG had the authority to make all final ethical decisions regarding the approval of narratives into the collection where the research team expressed uncertainty surrounding whether all inclusion criteria were met and/or whether an exclusion criterion was met. Moreover, the steering group could make recommended updates to the inclusion and exclusion criteria (see researchintorecovery.com/research/neon/neoncollection). This group was comprised of four LEAP members, a senior research fellow, and a research assistant. In total, there were eight meetings lasting 2 h per session with preparation for the meetings completed by an administrator; referred to as an Administrator, Professional, Managerial level 3 (APM3) within the host university. All the CSG meetings were necessary for the development of the NEON Intervention.

Once permission to use the stories were granted, the narratives were characterized using a standardized 77-item INCRESE tool (38). Researchers were trained to rate narratives to identify latent characteristics (e.g., the stage of recovery, genre) and manifest characteristics (e.g., narrator gender and content warnings). Each narrative was double rated for the content warning section of INCRESE. Training to code the recovery narratives using INCRESE was conducted as a part of a 10-day pilot study to test the validity of the tool, in which 100 narratives were rated. The cost of training researchers to use the INCRESE tool only accounts for the trainers' costs with the trainees' costs captured within the total coding time estimate.

For the process of coding the narratives, the researchers recorded the time taken to read, watch, or listen to the narratives. Although this provides an accurate estimate of the time taken to code the narratives, it does not contain a record of the length of breaks the researchers used during the process. Apart from rest periods being a standard part of a working day, a selection of the recovery stories may have caused an emotional or distressing impact on the researchers. Therefore, including breaks may more closely reflect the practical reality of coding the narratives (37). It was assumed that for every 6 h spent coding the coder had a 2-h break. After the recovery stories had been rated with INCRESE, they were added to the NEON Collection.

The NEON Intervention was delivered through a web-application to its users. A database specification document was required to support database implementation and source code development. Fundamental to the NEON Intervention is a recommender system that matches users with narratives intended to be of benefit. The researchers' time to conceptualize and develop the recommender system was challenging to estimate without records. The proposed budget line from the NEON trials was used as an informed approximation of the cost involved in developing the recommender system. Following this, the recommender system was integrated into the web-application codebase. As both developing and integrating the recommender system into the web-application are related tasks, they were combined into one cost.

An important component of developing the NEON Intervention was ensuring the web-application was appealing to use. Graphic designers made improvements to the aesthetical appeal of the web-application interfaces following feedback during the feasibility study and from LEAP members. The costs of keeping users engaged in the intervention (e.g., gamification, testimonials from other users, etc.) are spread between both graphic design and software development costs. Interaction design was also important to ensure the web-application's functions operated as intended (i.e., navigations).

Throughout the NEON study, task meetings were used to communicate and raise general discussions surrounding the interventions development. An assumption was made, through inspection of meeting agendas, that 33% of task meetings were necessary for the purpose of developing the intervention (i.e., excluding research costs). A total of 86 meetings were held with recorded attendance from members of the NEON team.

The international advisory board was used to provide expert consultation on safety strategies for the development of the NEON Collection and to provide advice on intervention engagement strategies. The meeting was attended by four professors and members of the NEON study team. During the intervention development stage, there was a single Intervention Design group meeting aimed at discussing particular features of the intervention and how they could be improved. This meeting was attended by two LEAP members and a senior research fellow. As before, all meetings had expenses covered.

To ensure that the evolving intervention operated as intended, the web-applications functions were tested by researchers using dummy accounts as well as data collection tools. This included testing of forms used to collect demographics and outcome data, and interactive features included to provide access to recovery narratives. During the analysis, the full intervention testing costs were deemed necessary for the development of the NEON Intervention on the basis that monitoring routine outcomes and usage data, necessary for the NEON trials, is commonplace in clinical practice (45).

Finally, a feasibility study to evaluate a prototype delivery of the NEON Intervention in a population of people with experience of mental health problems was conducted (37). The user feedback from this feasibility evaluation led to improvements in the intervention, for example, updating the color scheme to resemble the UK NHS website less closely. Digital technologies generally test a prototype of their technology on a sample of potential users for their feedback. Therefore, the full cost of the study was deemed relevant to development of NEON.

NEON Intervention delivery resource use

Table 6 summarizes resource use and unit cost categories and sources for the components required to deliver the NEON Intervention.

TABLE 6

Table 6. Resource use and unit cost input parameters for the delivery stage.

The web hosting capacity of the site, together with associated cyber-security features, is supplied by Amazon Web Services Lightsail (https://aws.amazon.com/lightsail/). This service allows websites to host a specific number of users for a publicly advertised price. The current intervention is designed to host 2,000 users. Therefore, a specific level of resource use was predetermined. The invoices from the supplier were made over a monthly billing period in US Dollars (USD); the exchange rates were determined by the credit card issuer.

The cost of personnel training for the administrator, within the host university, is referred to as Administrative, Professional and Managerial level 2 (APM2). The number of days spent training the administrator was recorded for each personnel conducting the training (see Table 5). Training costs would need to be incurred every time there is an administrative staff turnover. Therefore, an assumption was made that a new administrator would need to be trained every 1.5 years. Similarly, the cost of personnel training for the researchers who rated recovery narratives using the INCRESE tool is included as a delivery cost. It was assumed that training costs would need to be incurred every year.

The NEON Intervention required new narratives to be introduced into the collection over time to maintain diversity and relevance to users. Firstly, we consulted with the NEON team to estimate that an additional 200 narratives per year would be needed based on preliminary work looking at the diversity of the current NEON Collection. The cost per narrative was calculated from the current collection size of 659 recovery stories then re-scaled to 200 narratives. Although the cost per narrative approach can provide an estimate of the cost to update the narrative collection, it explicitly assumes a linear relationship between the cost and the narrative collection size. There is uncertainty about whether the cost of updating the narrative collection will increase or decrease for newly sourced narratives. In reality, the process of updating the narratives may become more streamlined and productivity gains can be made in coding the narratives. On the other hand, the cost of updating the narrative collection may be greater if sourcing new narratives becomes more cumbersome, e.g., exhausting the number of existing collections to source narratives.

The web-application requires ongoing maintenance to ensure the NEON Intervention can be delivered as intended for its users. There are challenges in costing for web-application maintenance due to the variability in need for maintenance and the broad definition of what maintenance means in practice. In this case, we define web-application maintenance as any change, modification, or update to the web-application codebase to correct faults, to improve performance, or to update the content on the web-application.

To deliver the NEON Intervention, it was assumed that 9.75 h per week of administrative support by an APM2 is required to conduct operational tasks (e.g., intervention engagement support tasks). The hours per week was estimated by assuming a smaller proportion (50%) of the administrative support observed during the NEON trials (19.5 h per week) would be required in a routine operational setting.

Operational activities that may have influenced the NEON Intervention's effectiveness (i.e., Advertising and Engagement strategies) were also considered to be a delivery cost. Advertisement for the study following agreed advertising principles (46) to recruit eligible participants for both NEON trials. Since the effectiveness and types of adverts may have impacted upon the effectiveness of the intervention it is included as a delivery cost. To ensure users of the NEON Intervention were making use of the intervention, engagement strategies (such as message prompting) were used to encourage use. As these strategies encourage the use of the intervention, they may impact the effectiveness of the intervention and are therefore a part of the delivery cost.

Throughout the delivery of the NEON Intervention, the NEON study was responsible for the wellbeing of those using the NEON Intervention. Safeguarding concerns were dealt with during the trials by the clinical principal investigator.

Sensitivity analysis plan

To assess the impact of our assumptions and parameter uncertainty, all input parameters were varied to their extreme values in one-way sensitivity analysis. To examine the impact of the structural assumptions, scenario analysis was used. The assumptions that were made during the base-case analysis for both the development and delivery of the NEON Intervention and the sensitivity and scenario analyses are shown in Table 7.

TABLE 7

Table 7. Sensitivity and scenario analysis plan.

Results

Costs of developing the NEON Intervention

In the base-case, the total number of hours to develop the intervention (excluding resource use external to the NEON team) was 2,709 h (45.2 days). The resource items that required the most personnel time were the staff time to code the narrative collection (1,092 h); the staff time to source recovery narratives (562 h); task meetings (341 h); and LEAP workshops (328 h). The members of staff that contributed the most hours include the coding researchers (1,092 h), the research assistants (895 h), and the senior research fellows (394 h). The cost per unit of staff time varied from £18.25 to £69.91 per hour.

A summary of the estimated costs of developing the NEON Intervention is provided in Table 8. The total cost of developing the NEON Intervention was £182,851. The largest cost components include software development (27%); LEAP workshops (23%); coding the narratives using the INCRESE tool (9%); and researchers' time to source narratives (9%). The total cost of curating the narrative collection was £82,710. The majority of this cost is attributed to the LEAP meetings (50%), coding the narratives using the INCRESE tool (20%), and the researchers' time to source narratives (19%). The total cost of developing the web-application was £64,067. The largest contributions to the development cost of the web-application were the software development (77%); developing the recommender system (12%); and the graphic design (7%).

TABLE 8

Table 8. Cost of developing the NEON Intervention (base-case).

Costs of delivering the NEON Intervention

In the base-case, the total number of personnel hours to deliver the intervention during the NEON trial was 1,708 h (28.5 days). In the NEON-O trial, the total number of personnel hours to deliver the intervention was 1,776 h (29.6 days). The resource items that require the most personnel time were the administrative support (54%); maintenance (19%); advertising (17%). The cost per unit of resource use varied between £2.40 and £125.

A summary of NEON Intervention delivery costs is provided in Table 9. The total cost of delivering the NEON Intervention during the NEON trial was £118,663 (£321 per user). In the NEON-O trial, the total delivery cost of the NEON Intervention was £123,444 (£241 per user). Therefore, the total delivery cost during the NEON trial was 4% lower than during the NEON-O trial. However, the cost per user during the NEON trial was 33% higher than the NEON-O trial. The proportion of fixed costs (advertising, engagement, and safeguarding) was 22% during the NEON trial compared to 21% during the NEON-O trial. The largest cost components include updating the narrative collection (50%); advertising (19%); administration (14%); and software maintenance (11%). The cost of delivering the intervention for a 1-year period is £68,521.

TABLE 9

Table 9. Cost of delivering the NEON Intervention (base-case).

Sensitivity analysis

The one-way sensitivity analysis of the most sensitive cost components for NEON Intervention development are presented as a Tornado diagram in Figure 1. Uncertainty in the cost of the LEAP meetings had the greatest impact on the base-case estimate. Specifically, the total cost may be 5.8% higher or lower than the base-case estimate. The difference between the extreme values of the LEAP meetings, task meetings and recommender system development are £21,041; £17,910; and £11,513, respectively. Varying cost components such as intervention testing and efforts to secure ethical approval had a relatively smaller effect on the total development cost. By varying these costs to their extreme values, the impact on total cost of intervention testing and effort to secure ethical approval was +/- £1,296 and +/-£1,003, respectively. Other components had comparatively little effect on the overall development cost such as the database specification (+/- £150).

FIGURE 1

Figure 1. One-way sensitivity analysis - NEON Intervention development costs.

The one-way sensitivity analysis of the cost components for the NEON Intervention delivery during the NEON trial are presented as a Tornado diagram in Figure 2. Uncertainty in the cost of administration was shown to have the largest effect on the base-case estimate. Specifically, the total cost may be 7% higher or lower than the base-case estimate. Cost components such as the web-application maintenance and updating the narrative collection had a relatively smaller effect on delivery costs. For example, the impact on the total delivery cost of varying web-application maintenance to its extreme values was £6,534 higher or lower than the base case estimate. Cost components such as engagement had a relatively small effect on the delivery of the NEON Intervention during the NEON trial. The impact on the total delivery cost of varying the engagement component to its extreme values was 0.5% higher or lower than the base case estimate. Similar results can be seen in the one-way sensitivity analysis of the cost components for the NEON Intervention delivery during the NEON-O trial (see Figure 3).

FIGURE 2

Figure 2. One-way sensitivity analysis - NEON Intervention delivery costs during the NEON trial.

FIGURE 3

Figure 3. One-way sensitivity analysis - NEON Intervention delivery costs during the NEON-O trial.

The scenario analysis of the structural assumptions for developing the NEON Intervention are presented as a Tornado diagram in Figure 4. The impact on the total cost of the best and worst case scenario analyses show a feasible total cost range to develop the NEON Intervention. The cost of developing the NEON Intervention given the best possible scenario is £54,724 lower than the base-case estimate of £181,851. Similarly, given the worst possible scenario, the cost of developing the NEON Intervention is £60,805 higher than the base-case estimate.

FIGURE 4

Figure 4. Scenario analysis - NEON Intervention development costs.

The scenario analysis of the structural assumptions for the NEON Intervention delivery during the NEON trial are presented in Figure 5. The best-case scenario is £39,213 lower than the base-case estimate, and the worst-case scenario cost is £71,180 higher than the base-case estimate. Similar results can be seen in the scenario analysis of the structural assumptions for the NEON Intervention delivery during the NEON-O trial (see Figure 6).

FIGURE 5

Figure 5. Scenario analysis - NEON Intervention delivery costs during the NEON trial.

FIGURE 6

Figure 6. Scenario analysis - NEON Intervention delivery costs during the NEON-O trial.

As expected, increasing the number of users, reduced the cost per user year, such that 500, 1,000, or 2,000 users cost £137.04, £68.52, £34.26, respectively.

Discussion

Key findings

We identified several resource use categories for intervention development and delivery through a literature review and work with experts on the research team. Developing the NEON Intervention cost £182,851, which was largely attributed to building the NEON Collection (£82,710) and developing the web-application (£64,067). The curation costs were mostly made up of LEAP meetings (50%); narrative coding using the INCRESE tool (20%); and the researchers' time to source recovery narratives (19%). The largest components of the web-application costs included software development (77%); developing the recommender system (12%); and the graphic design costs (7%).

Delivering the NEON Intervention during the NEON trial and NEON-O trial costed £118,663 and £123,444, respectively. This equates to £349 (NEON trial) and £241 (NEON-O trial) per user. Over an annual period, the NEON Intervention cost was £68,521. Delivery costs were driven by updating the narrative collection (50%); advertisement (19%); and administrative support. Similarly to other studies (20), we found that total costs are dependent on usage rates, which are difficult to predict. Due to a high proportion of fixed costs, costs per user were high for the lower caseload in NEON-O but would be expected to decline with increasing volume of use.

Strengths and limitations

The strength of our study is the use of published guidelines where available to inform our costing methods. We have provided detailed and transparent reporting of cost components and we have utilized multi-disciplinary input into identifying categories, including strong service user involvement. We have carried out sensitivity analysis to make explicit which parameters are associated with uncertainty, and the direction and magnitude of that uncertainty.

The limitations of our study include the necessary use of expert opinion sources where primary data were not collected or available. Due to limitations in patient-level resource use data availability, we have had to use top-down costing methods rather than bottom-up, or micro-costing approaches, which arguably does not provide a patient-level cost sensitive to different levels of usage by individual patients. This limitation is common to costing DHIs, due to the nature of the costs involved (47).

Defining a starting point for development costs necessarily meant excluding early development costs. As with other research-led endeavors, this work is supported by the accumulation of an existing knowledge base and the experience and expertise of the research team, as well as the opportunity cost of the researchers. This can be conceptualized using the health service research payback model which examines the complex interaction and costs of early and late publicly-funded research, and their effect on research outputs (48). There has been a substantial program of earlier research to support the development and testing of the NEON Intervention, which is outside the scope of this work to quantify. Other costs which were outside the scope include the intellectual work involved in the development of the NEON proposal for funding, the other types of resource such as existing collaboration networks which could be accessed for advice, and work conducted in other studies in the research group which may have cross-fertilized the NEON work. We also did not include the RCT costs in our development costs as we explicitly examined development costs up to the point of the beginning of the RCTs. The impact that the trial's findings will have upon the NEON Intervention is currently uncertain. As such, it is unclear what changes the NEON trials may inform to improve, scale, or discontinue the intervention in its current form. Therefore, trial costs were not considered as a development cost during the current development period.

The delivery costs provided here are derived from the delivery of the NEON Intervention during the trials, so it is likely that delivery costs in practice will be different, and health care payers need to be prepared for costs to vary once the intervention is implemented outside a clinical trial environment. How the NEON Intervention is implemented in the NHS will likely determine what resources will be needed in future versions of the intervention. As the NEON trials do not compare the NEON Intervention with a face-to-face version of the same intervention, we are not able to examine the differences in costs (or effectiveness) between these two scenarios.

Assumptions had to be made around tasks and roles taken up by researchers in the trial that would actually be covered by healthcare professionals, administrators, and other members of the healthcare team when the intervention is delivered in practice. Necessarily there are likely to be some differences between trial delivery and practice delivery. For example, the clinical principal investigator coordinated all safeguarding activities during NEON Intervention delivery as part of the RCT. If the NEON Intervention is rolled out in practice, at a substantial scale, then the safeguarding approaches and infrastructure needed will necessarily be different. Safeguarding approaches will need to scale with safeguarding demands, and to take into account different regulatory requirements in everyday practice compared to clinical trials. We have treated advertising costs in the trials as a proxy for accessing patients in practice. Recruitment for the NEON Trial was targeted carefully, for example through the production of several 100 online messages with content specific to a psychosis trial and the design and dissemination of targeted adverts (for example displayed as banners on websites used by health and social care professionals and potential participants). This involved human effort and expertise to design these materials. If the NEON Intervention is rolled out to a general mental health population on a larger scale through online mechanisms, then less targeting of recruitment material might be needed, and individual messages or adverts can also be reused and redeployed. This means that the human cost of generating recruitment material might be less and the dominant cost might be spent on the services that social media companies provide to promote messages to a relevant audience. Especially at the point where the NEON Intervention is deployed on a wider scale, the cost of recruiting one person through social media promotion should be routine to estimate.

Implementation of the NEON Intervention beyond the trials will lie on a spectrum. On the one end, the implementation through statutory mental health services (costs associated with staff awareness and training, implementing safeguarding procedures etc.), through to the implementation via primary care or the voluntary sector, to “direct to consumer” (costs associated with advertising), and the value for money offered by these different implementation routes (which may be separate or additive) that could be evaluated in relation to reach and engagement. The most efficient way to implement DHIs like the NEON Intervention is very relevant in a resource-constrained system as implementation methods cost money and affect the effectiveness and cost-effectiveness of the DHI (49).

Other limitations in the delivery costs include the lack of regulatory or scale-up costs, such that these costs may be an underestimate of true costs once the NEON Intervention is implemented in practice. The derivation of costs per user from top-down costs necessarily means that the cost per user is related to the number of users assumed in the calculations. The number of users could be the number of people in each trial, the number of people eligible to use the NEON Intervention in the real world (21), or we would suggest, the maximum capacity of the current technology implementation to provide an appropriate level of quality of service (50) to its users (for example with sufficient responsiveness of interactive features to allow for a satisfying engagement). If we use the number of people in each trial to estimate cost per user, the mean cost per user could be overestimated as the maximum technological capacity consistent with acceptable service quality has not been reached. However, using the number of people eligible to use the NEON Intervention in the real world is not straightforward. This is for a number of reasons: the computational complexity of the algorithms used in the technology may not scale linearly with the size of the user base (meaning that needed server performance and hence server cost may not scale linearly), and at discrete points, technology re-engineering may be needed to maintain service quality, for example by replacing propriety content delivery system with commercially-provided Content Delivery Networks (51). This reflects the development cycle for web-based DHIs. It is similar to most start-ups that proceed through a series of versions of their systems, re-engineering them each time for a larger number of users. Costs associated with capacity-related engineering and enhanced server capacity should be spread across the anticipated number of users, and allocated equally per user. For example, additional costs required to scale the intervention from 2,000 to 5,000 users should be spread equally across those 3,000 extra users to prevent spikes in costs per user. In the final report for the NEON trials, we will examine this issue further and look at scenarios for future costs for the NEON Intervention as it is scaled up to a range of anticipated user base sizes.

Given the evolving nature of DHIs, input parameters into economic evaluations (like delivery costs) need to be re-examined at different stages of the intervention capacity. It is likely that certain cost components will change over time. Hosting costs are likely to reduce due to increases in process performance and reductions in storage cost per unit (Moore's Law) (52), and salaries may increase (given human resources have quite a large impact on the delivery, this is important).

Another issue beyond the scope of this study is the likelihood that the intervention will be delivered differently once it is implemented in practice. This will have an impact not only on resource use but also on effectiveness of the intervention. However, the data we have provided will provide healthcare providers with approximations of the resources required to deliver the intervention in practice.

Comparisons with existing evidence

We are not aware of a DHI that has focused on the use of narratives to support recovery in mental health, so it is not possible to compare our development and delivery costs with another equivalent intervention or directly relevant costing study. It is also unclear as to the utility of comparing the costs we have estimated with other mental health DHIs costs, due to the lack of comparability of costing methods between studies. The reporting of different development and delivery cost components of mental health DHIs varies, making meaningful comparison difficult (11, 15).

Development costs are reported in a minority of economic evaluations, as they are usually seen as sunk costs and not relevant to the health and social care provider. The list price of commercially developed healthcare products such as pharmaceuticals are perceived to include development costs and thus recoup those costs. Development costs are more difficult to determine for DHIs. Opinions differ on the inclusion of development costs, and it is the perspective that should determine whether they are included or not (47). When development costs are included, a judgement has to be made regarding when development costs are considered to begin and end, as we had to make in our study, and this judgment can have a significant effect on the overall costs derived.

When reported, development costs vary from as little as £19,000 (53) to £500,000 (54), but whether this variation stems from the varied nature of the DHIs or from the methods used to collect resource use data is challenging to untangle. However, they can influence whether development occurs, so explicit methods for deciding on resource categories to include, methods of resource use data collection and sources of unit costs should be explicitly given in any study. We used a similar approach to Joshi and colleagues who reported explicit methods to derive development cost for a digital program for training community health workers to deliver treatment for depression in rural India (22). Similarly to our study, they reported that staffing costs constituted 61% of development costs.

In a recent review of methods used in economic evaluations of mental health DHIs, 16 out of 66 did not report staffing costs as part of delivery costs (15). Given the human resource intensive nature of DHI delivery as found in our study, this omission would lead to a significant underestimate of costs. Only 14 out of 66 studies included costs for website maintenance and hosting, again we found that this constituted a significant delivery cost.

We used the NICE guidance to support our costing methodology. However, there have been questions raised surrounding how fit for purpose this guidance is for costing DHIs in practice (21). Like other economic evaluation guidelines, although they provide a costing framework, there is little practical guidance on specific costing (47). We identified several resource use categories in intervention development and delivery, and future developers of DHIs should consider including these categories, and working with relevant experts to identify any further categories specific to their intervention. We also recommend keeping formal resource use records to allow easier derivation of costs. Since DHIs (in keeping with commercial web-applications) are typically subject to ongoing periods of maintenance punctuated by periodic redevelopment and reengineering work which will have an impact on the cost per user, we would advocate for the collection of a broad range of case studies providing evidence on the cost over the life of a DHI of ongoing development work. We have excluded conceptual work conducted before the NEON program was funded, and in the early stages of the NEON program. Future studies might consider approaches to costing in such activities.

Human resources constituted a large proportion of our development and delivery costs. These estimates were affected by uncertainties around which costs to include, other than salary, so we followed the PSSRU approach to include employers' costs, as well as estates and indirect costs for the organization employing that person (55). Our sensitivity analysis demonstrated that inclusion of these latter costs had a significant effect on overall costs.

Conclusion

This study makes two knowledge contributions. First, it provides a usable estimate of the cost of developing and implementing a DHI from software and intervention development, through feasibility testing and up to commencement of the definitive trial. This can be used to inform commissioning of new DHIs in general, giving explicit consideration to all the different types of resources required, and the quantity and cost, as well as specifically, the implementation of the NEON Intervention. Second, the costing challenges which have been identified indicate the need for updated best practice guidance for economic evaluation of DHIs by NICE and other clinical and funding agencies.

Relevance for clinical practice

The NEON Intervention is intended for widespread use as a low-cost self-management intervention. Two uses are envisaged: adjunctive to clinical treatment and direct access. This study identifies the costs associated with population-level roll-out of the NEON Intervention. In relation to use within services, the staff costs needed to support access are identified. In relation to direct access, the public health costs associated with maintaining and developing the intervention are estimated. We are currently evaluating the effectiveness of the NEON intervention based on the two RCTs described in this study. Data analysis is near completion and will be published separately building on the findings in this study. These findings will inform decision-making about whether, and how, to implement the NEON Intervention at scale.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Leicester Central Research Ethics Committee, 19/EM/0326. Informed consent to participate was obtained from all participants using an online form. The patients/participants provided their written informed consent to participate in this study.

Author contributions

RE, SG, LP, and SR-E contributed to the conception and design of the study. LP carried out the data collection and analysis and wrote the first draft of the manuscript. SR-E, SG, and RE wrote sections of the manuscript. JL-B, FN, CB, AG, JN, DQ, and SB contributed to the data collection. JL-B and FN provided feedback on the analysis. MS is the chief investigator for this trial. RE is the lead health economist. All authors have read, commented, and reviewed the manuscript.

Funding

This study was funded by the NIHR [Personal experience as a recovery resource in psychosis: Narrative Experiences Online (NEON) Programme (RP-PG-0615-20016)]. The funder was not involved in the development of the methods, data collection, analysis or writing of this manuscript.

Acknowledgments

MS acknowledges the support of NIHR Nottingham Biomedical Research Center.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.1028156/full#supplementary-material

References

1. Murphy M, Scott LJ, Salisbury C, Turner A, Scott A, Denholm R, et al. Implementation of remote consulting in UK primary care following the COVID-19 pandemic: a mixed-methods longitudinal study. Br J General Practice. (2021) 71:e166–77. doi: 10.3399/BJGP.2020.0948

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Neves AL, Li E, Serafini A, Jimenez G, Lingner H, Koskela TH, et al. Evaluating the impact of COVID-19 on the adoption of virtual care in general practice in 20 countries (inSIGHT): protocol and rationale study. JMIR Res Protoc. (2021) 10:e30099. doi: 10.2196/30099

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. (2012) 32:329–42. doi: 10.1016/j.cpr.2012.02.004

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Himle JA, Weaver A, Zhang A, Xiang X. Digital mental health interventions for depression. Cogn Behav Pract. (2022) 29:50–9. doi: 10.1016/j.cbpra.2020.12.009

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual Research Review: digital health interventions for children and young people with mental health problems–a systematic and meta-review. J Child Psychol Psychiatry. (2017) 58:474–503. doi: 10.1111/jcpp.12663

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Simblett S, Birch J, Matcham F, Yaguez L, Morris R. A systematic review and meta-analysis of e-mental health interventions to treat symptoms of posttraumatic stress. JMIR Ment Health. (2017) 4:e14. doi: 10.2196/mental.5558

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Fortuna KL, Naslund JA, LaCroix JM, Bianco CL, Brooks JM, Zisman-Ilani Y, et al. Digital peer support mental health interventions for people with a lived experience of a serious mental illness: systematic review. JMIR Ment Health. (2020) 7:e16460. doi: 10.2196/16460

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. Oxford: Oxford University Press (2015). ISBN: 0191643580.

Google Scholar

9. Topooco N, Riper H, Araya R, Berking M, Brunn M, Chevreul K, et al. Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Intervent. (2017) 8:1–9. doi: 10.1016/j.invent.2017.01.001

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic evaluations of Internet interventions for mental health: a systematic review. Psychol Med. (2015) 45:3357–76. doi: 10.1017/S0033291715001427

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Mitchell LM, Joshi U, Patel V, Lu C, Naslund JA. Economic evaluations of internet-based psychological interventions for anxiety disorders and depression: a systematic review. J Affect Disord. (2021) 284:157–82. doi: 10.1016/j.jad.2021.01.092

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Paganini S, Teigelkotter W, Buntrock C, Baumeister H. Economic evaluations of internet- and mobile-based interventions for the treatment and prevention of depression: a systematic review. J Affect Disord. (2018) 225:733–55. doi: 10.1016/j.jad.2017.07.018

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Kass AE, Balantekin KN, Fitzsimmons-Craft EE, Jacobi C, Wilfley DE, Taylor CB. The economic case for digital interventions for eating disorders among United States college students. Int J Eat Disord. (2017) 50:250–8. doi: 10.1002/eat.22680

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Buntrock C, Kählke F, Smit F, Ebert DD. A systematic review of trial-based economic evaluations of internet- and mobile-based interventions for substance use disorders. Eur J Public Health. (2021) 31(Suppl.1):i19-28. doi: 10.1093/eurpub/ckz022

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Jankovic D, Saramago Goncalves P, Gega L, Marshall D, Wright K, Hafidh M, et al. Cost effectiveness of digital interventions for generalised anxiety disorder: a model-based analysis. PharmacoEconomics. (2021) 21:318. doi: 10.1007/s41669-021-00318-y

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Bergmo TS. How to measure costs and benefits of ehealth interventions: an overview of methods and frameworks. J Med Internet Res. (2015) 17:e254. doi: 10.2196/jmir.4521

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Neher M, Nygårdh A, Broström A, Lundgren J, Johansson P. Perspectives of policy makers and service users concerning the implementation of eHealth in Sweden: interview study. J Med Internet Res. (2022) 24:e28870. doi: 10.2196/28870

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Lennon MR, Bouamrane M-M, Devlin AM, O'Connor S, O'Donnell C, Chetty U, et al. Readiness for delivering digital health at scale: lessons from a longitudinal qualitative evaluation of a national digital health innovation program in the United Kingdom. J Med Internet Res. (2017) 19:e42. doi: 10.2196/jmir.6900

PubMed Abstract | CrossRef Full Text | Google Scholar

19. NICE. Evidence Standards Framework (ESF) for Digital Health Technologies. National Institute for Health and Care Excellence (2018). Available online at: https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies (accessed July 12, 2022).

Google Scholar

20. Huter K, Krick T, Rothgang H. Health economic evaluation of digital nursing technologies: a review of methodological recommendations. Health Econ Rev. (2022) 12:35. doi: 10.1186/s13561-022-00378-8

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Gomes M, Murray E, Raftery J. Economic evaluation of digital health interventions: methodological issues and recommendations for practice. Pharmacoeconomics. (2022) 40:367–78. doi: 10.1007/s40273-022-01130-0

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Joshi U, Naslund JA, Anand A, Tugnawat D, Vishwakarma R, Bhan A, et al. Assessing costs of developing a digital program for training community health workers to deliver treatment for depression: a case study in rural India. Psychiatry Res. (2022) 307:114299. doi: 10.1016/j.psychres.2021.114299

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Llewellyn-Beardsley J, Rennick-Egglestone S, Callard F, Crawford P, Farkas M, Hui A, et al. Characteristics of mental health recovery narratives: Systematic review and narrative synthesis. PLoS ONE. (2019) 14:e0214678. doi: 10.1371/journal.pone.0214678

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Llewellyn-Beardsley J, Rennick-Egglestone S, Bradstreet S, Davidson L, Franklin D, Hui A, et al. Not the story you want? Assessing the fit of a conceptual framework characterising mental health recovery narratives. Social Psychiatry Psychiatric Epidemiol. (2020) 55:295–308. doi: 10.1007/s00127-019-01791-x

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Rennick-Egglestone S, Morgan K, Llewellyn-Beardsley J, Ramsay A, McGranahan R, Gillard S, et al. Mental health recovery narratives and their impact on recipients: systematic review and narrative synthesis. Can J Psychiatry. (2019) 2019:706743719846108. doi: 10.1177/0706743719846108

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Rennick-Egglestone S, Ramsay A, McGranahan R, Llewellyn-Beardsley J, Hui A, Pollock K, et al. The impact of mental health recovery narratives on recipients experiencing mental health problems: qualitative analysis and change model. PLoS ONE. (2019) 14:e0226201. doi: 10.1371/journal.pone.0226201

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Ng F, Charles A, Pollock K, Rennick-Egglestone S, Cuijpers P, Gillard S, et al. The mechanisms and processes of connection: developing a causal chain model capturing impacts of receiving recorded mental health recovery narratives. BMC Psychiatry. (2019) 19:2405. doi: 10.1186/s12888-019-2405-z

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Ng F, Newby C, Robinson C, Llewellyn-Beardsley J, Yeo C, Roe J, et al. How do recorded mental health recovery narratives create connection and improve hopefulness? J Mental Health. (2022) 31:273–80. doi: 10.1080/09638237.2021.2022627

PubMed Abstract | CrossRef Full Text | Google Scholar

29. McGranahan R, Rennick-Egglestone S, Ramsay A, Llewellyn-Beardsley J, Bradstreet S, Callard F, et al. Curation of mental health recovery narrative collections: systematic review and qualitative synthesis. JMIR Mental Health. (2019) 6:e14233. doi: 10.2196/14233

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Yeo C, Hare-Duke L, Rennick-Egglestone S, Bradstreet S, Callard F, Hui A, et al. The VOICES typology of curatorial decisions in narrative collections of the lived experiences of mental health service use, recovery, or madness: qualitative study. JMIR Mental Health. (2020) 7:e16290. doi: 10.2196/16290

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Yeo C, Rennick-Egglestone S, Armstrong V, Borg M, Franklin D, Klevan T, et al. Uses and misuses of recorded mental health lived experience narratives in healthcare and community settings: systematic review. Schizophr Bull. (2022) 48:134–44. doi: 10.1093/schbul/sbab097

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Slade M, Rennick-Egglestone S, Blackie L, Llewellyn-Beardsley J, Franklin D, Hui A, et al. Post-traumatic growth in mental health recovery: qualitative study of narratives. BMJ Open. (2019) 9:e029342. doi: 10.1136/bmjopen-2019-029342

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Ng F, Ibrahim N, Franklin D, Jordan G, Lewandowski F, Fang F, et al. Post-traumatic growth in psychosis: a systematic review and narrative synthesis. BMC Psychiatry. (2021) 21:3. doi: 10.1186/s12888-021-03614-3

PubMed Abstract | CrossRef Full Text | Google Scholar

34. McGranahan R, Jakaite Z, Edwards A, Rennick-Egglestone S, Slade M, Priebe S. Living with psychosis without mental health services: a narrative interview study. BMJ Open. (2021) 11:e045661. doi: 10.1136/bmjopen-2020-045661

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Hui A, Rennick-Egglestone S, Franklin D, Walcott R, Llewellyn-Beardsley J, Ng F, et al. Institutional injustice: implications for system transformation emerging from the mental health recovery narratives of people experiencing marginalisation. PLoS ONE. (2021) 16:e0250367. doi: 10.1371/journal.pone.0250367

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Roe J, Brown S, Yeo C, Rennick-Egglestone S, Repper J, Ng F, et al. Opportunities, enablers, and barriers to the use of recorded recovery narratives in clinical settings. Front Psychiatry. (2020) 11:589731. doi: 10.3389/fpsyt.2020.589731

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Slade M, Rennick-Egglestone S, Llewellyn-Beardsley J, Yeo C, Roe J, Bailey S, et al. Recorded mental health recovery narratives as a resource for people affected by mental health problems: development of the narrative experiences online (NEON) intervention. JMIR Format Res. (2021) 5:e24417. doi: 10.2196/24417

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Llewellyn-Beardsley J, Barbic S, Rennick-Egglestone S, Ng F, Roe J, Hui A, et al. INCRESE: development of an inventory to characterize recorded mental health recovery narratives. J Recovery Ment Health. (2020) 3:25–44. Available online at: https://jps.library.utoronto.ca/index.php/rmh/article/view/34626

PubMed Abstract | Google Scholar

39. Helsper EJ. A corresponding fields model for the links between social and digital exclusion. Commun Theory. (2012) 22:403–26. doi: 10.1111/j.1468-2885.2012.01416.x

CrossRef Full Text | Google Scholar

40. Rennick-Egglestone S, Elliott R, Smuk M, Robinson C, Bailey S, Smith R, et al. Impact of receiving recorded mental health recovery narratives on quality of life in people experiencing psychosis, people experiencing other mental health problems and for informal carers: Narrative Experiences Online (NEON) study protocol for three randomised controlled trials. Trials. (2020) 21:661. doi: 10.1186/s13063-020-04428-6

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Rennick-Egglestone S, Elliott R, Newby C, Robinson C, Slade M. Impact of receiving recorded mental health recovery narratives on quality of life in people experiencing non-psychosis mental health problems (NEON-O Trial): updated randomised controlled trial protocol. Trials. (2022) 23:6027. doi: 10.1186/s13063-022-06027-z

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Chapel JM, Wang G. Understanding cost data collection tools to improve economic evaluations of health interventions. Stroke Vasc Neurol. (2019) 4:214. doi: 10.1136/svn-2019-000301

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Eschenbach TG. Technical note: constructing tornado diagrams with spreadsheets. Eng Economist. (2006) 51:195–204. doi: 10.1080/00137910600695676

CrossRef Full Text | Google Scholar

44. Husereau D, Drummond M, Augustovski F, De Bekker-Grob E, Briggs AH, Carswell C, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. Br Med J. (2022) 2022:e067975. doi: 10.1136/bmj-2021-067975

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychother Res. (2015) 25:6–19. doi: 10.1080/10503307.2013.817696

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Rennick-Egglestone S. Principles for the production and dissemination of recruitment material for three clinical trials of an online intervention. Trials. (2021) 22:4. doi: 10.1186/s13063-021-05412-4

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Jacobs JC, Barnett PG. Emergent challenges in determining costs for economic evaluations. Pharmacoeconomics. (2017) 35:129–39. doi: 10.1007/s40273-016-0465-1

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Buxton M, Hanney S. How can payback from health research be assessed? J Health Serv Res Policy. (1996) 1:35–43. doi: 10.1177/135581969600100107

CrossRef Full Text | Google Scholar

49. Fenwick E, Claxton K, Sculpher M. The value of implementation and the value of information: combined and uneven development. Med Decis Making. (2008) 28:21–32. doi: 10.1177/0272989X07308751

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Zhao W, Olshefski, D, Schulzrinne, HG,. Internet Quality of Service: An Overview. Columbia University Academic Commons. (2000). Availabole online at: https://academiccommons.columbia.edu/doi/10.7916/D8T15FTV (accessed January 10, 2022).

Google Scholar

51. Zolfaghari B, Srivastava G, Roy S, Nemati HR, Afghah F, Koshiba T, et al. Content delivery networks: state of the art, trends, and future roadmap. ACM Comput Surv. (2020) 53:34. doi: 10.1145/3380613

CrossRef Full Text | Google Scholar

52. Mack CA. Fifty years of Moore's law. IEEE Trans Semiconduct Manufact. (2011) 24:202–7. doi: 10.1109/TSM.2010.2096437

CrossRef Full Text | Google Scholar

53. Lobban F, Dodd AL, Sawczuk AP, Asar O, Dagnan D, Diggle PJ, et al. Assessing feasibility and acceptability of web-based enhanced relapse prevention for bipolar disorder (ERPonline): a randomized controlled trial. J Med Internet Res. (2017) 19:e85. doi: 10.2196/jmir.7008

PubMed Abstract | CrossRef Full Text | Google Scholar

54. McNamee P, Murray E, Kelly MP, Bojke L, Chilcott J, Fischer A, et al. Designing and undertaking a health economics study of digital health interventions. Am J Prev Med. (2016) 51:852–60. doi: 10.1016/j.amepre.2016.05.007

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Jones KC,. Burns, Amanda. Unit Costs of Health Social Care 2021. (Unit Costs of Health Social Care). ISBN: 9781911353140. PSSRU; University of Kent (2021). Available online at: https://www.pssru.ac.uk/project-pages/unit-costs/unit-costs-of-health-and-social-care-2021/~

Google Scholar