Internet-Delivered Cognitive Behavior Therapy as a Prequel to Face-To-Face Therapy for Depression and Anxiety: A Naturalistic Observation

Background: The UK’s Improving Access to Psychological Therapies (IAPT) program is a stepped-care model treating individuals with depression and anxiety disorders. Internet-delivered cognitive behavioral therapy (iCBT) is routinely offered to individuals with mild to moderate symptoms, but its applicability to individuals with severe clinical symptoms and requiring a high-intensity intervention is relatively unknown. The current study sought to investigate the potential impacts of using iCBT as a prequel for patients requiring high-intensity treatment (HIT; face-to-face) for depression and anxiety in IAPT.

Methods: The study utilized an open study design. One hundred and twenty-four participants who were on a waiting list for high-intensity, face-to-face psychological treatment were offered iCBT. Psychometric data on symptoms of depression, anxiety, and functioning were collected from participants before starting and on finishing iCBT and at the point of service exit. Therapeutic alliance data were collected from patients and clinicians during treatment. Patient pathway data, such as number of treatment sessions and time in treatment, was also collected and incorporated into the analysis.

Results: Significant reductions across primary outcome measures of depression and anxiety, as well as improved functioning, were observed from baseline to iCBT treatment exit, and from iCBT exit to service exit. Analysis of the therapeutic alliance data for patients and clinicians illustrated differences in outcome for those who dropped out and those who completed treatment.

Discussion: This study illustrates the potential for using iCBT as a prequel to high-intensity therapy for depression and anxiety disorders and is the first of its kind to do so within IAPT stepped care. The results show that iCBT is a valuable option reducing waiting times and enhancing clinical efficiency. The study contributes to the well-established evidence on online psychological treatments worldwide, but further clinical and service development research is necessary to scale these treatments appropriately.

Introduction

According to the Global Burden of Disease study, depression and anxiety disorders contribute the greatest degree of disability amongst all mental and substance abuse disorders (1). Psychological therapies have been shown to be both clinically and cost effective in the treatment and management of depression and anxiety disorders (2) and preferred by some over pharmacological interventions (3). In recent years, technology has facilitated the dissemination of psychological therapy, particularly cognitive behavior therapy, through internet-delivered interventions for depression and anxiety (4). Based on the available evidence for the efficacy of internet-delivered cognitive behavioral therapy (iCBT) for depression and anxiety across multiple meta-analyses (5–7), these interventions are currently deployed in a supported manner as part of routine care in mental health clinics in several countries (8). The literature shows that clients value the helpfulness of supporters, as they encourage and motivate clients to keep using the intervention as well as provide helpful guidance and feedback, which contribute to enhanced outcomes (9, 10, 11) Furthermore, iCBT is included as a treatment option in the UK clinical guidelines for the treatment of depression and anxiety disorders (12, 13).

Different authors advocate for the inclusion of iCBT interventions within stepped mental healthcare models (14, 15). Stepped-care models seek to match treatment intensity to client needs by providing the least intrusive and most effective intervention for the client upon entering services (16). The Improving Access to Psychological Therapies (IAPT) program is a stepped-care approach to psychological care for people with depression and anxiety within the National Health Service (NHS) in the UK (17). Specifically, the IAPT model delivers low-intensity psychological interventions (i.e. iCBT, guided self-help) (18) at step 2 (mild to moderate presentations of depression and anxiety disorders) and delivers high- intensity psychological interventions at step 3 (severe presentations of depression and anxiety disorders).

In the IAPT model, supporters are primarily psychological wellbeing practitioners (PWPs), graduate workers who are trained to provide low-intensity mental health services in primary care (18). PWPs are trained to identify and assess common presentations of mental health difficulties and work collaboratively with their clients to develop a treatment plan that best suits their needs (19) PWPs, as they encourage and motivate clients to keep using the intervention as well as provide helpful guidance and feedback, which contribute to enhanced outcomes (15–17). The work of the PWP is principally guided by the Reach Out Curriculum and advocates competencies in six core areas, namely, information gathering, information giving, shared decision-making, low-intensity treatment interventions, supervision and values, culture, diversity, and policy (19).

The deployment of iCBT as part of both routine care in mental health clinics and step 2 of IAPT is similar in that these interventions are offered to individuals with mild to moderate symptoms of depression and anxiety and generally exclude those with more severe presentations. Thus, most studies testing the efficacy of iCBT have focused on individuals with mild to moderate symptoms of depression and anxiety, leading to a less established evidence base for the use of iCBT with more severe presentations (20). Despite this, studies that have explored the effects of iCBT on those with severe presentations have illustrated positive effects (21, 22), even showing maintenance of improvements at follow-up (22, 23). A recent meta-analysis of studies of iCBT robustly demonstrates the relevance of iCBT for severe depression presentations; the authors argue that their findings should lay to rest the notion that iCBT should be limited to clients with milder depressions (7). This raises the possibility of using iCBT as a frontline intervention alongside high-intensity therapy (HIT) service provision for individuals with severe symptoms of depression and anxiety (22). In the IAPT stepped-care context, iCBT could be offered while service users wait for high-intensity treatment resources (face-to-face) to become available. To date, however, there are no data on the utility of iCBT as part of HIT service delivery in IAPT.

Therapeutic alliance refers to the collaborative nature of the interaction between therapist and client that emerges due to the affective bond between them, the agreement on the specific tasks in treatment, and the agreement on therapeutic goals (24, 25). Psychotherapy research findings indicate that alliance is related to therapeutic change (26–28) and overall satisfaction with care (29). In the field of internet interventions, research on therapeutic alliance with online supporters is scarce (30). A recent narrative review concluded that a positive therapeutic alliance can be formed in guided iCBT, with client ratings similar to those found in face-to-face treatment (31). This review also suggests that while an alliance may be formed online, the bond may be less important for the alliance to develop through this medium than in face-to-face therapy. Another review concluded that the therapeutic alliance in iCBT, as traditionally measured, has shown mixed results (32). There is opportunity to explore further the nature of the therapeutic alliance online and its relative contribution to outcomes. Given the established importance of the relationship between client and supporter in alliance and outcomes, it could be valuable to consider both client and clinician ratings to provide a comprehensive picture and inform the development and accurate deployment of iCBT as part of stepped care (30, 33–35). Also, some authors suggest that there is a need for studies that measure therapeutic alliance ratings at different points of treatment, as this could provide important information about alliance development (36).

The current study sought to examine clinical outcomes and the alliance online in delivering the SilverCloud iCBT for depression and anxiety disorders as a prequel to face-to-face therapy for clients with severe symptoms of depression and anxiety. Based on previous research of the SilverCloud platform and programs (37–39), the following research questions were developed:

1. Can clinical symptoms of severe depression/anxiety improve pre–post iCBT?

2. Do patients with severe depression/anxiety who complete iCBT continue to improve with subsequent face-to-face therapy?

3. Do patients with severe depression/anxiety develop a therapeutic alliance with the practitioner who supports them during iCBT, and is this maintained over time?

4. What do clinicians think about iCBT as a prequel to face-to-face therapy for patients with severe depression/anxiety?

Materials and Methods

Design

The study followed an uncontrolled feasibility design, On the one hand, it aimed at examining quantitatively clinical outcomes on depression and anxiety, functional outcomes in terms of work and social functioning, waiting time reduction, and therapeutic alliance between client and clinician in regards to step 3 services. Qualitatively, clinicians’ experiences about the acceptability of the online intervention as a prequel to high-intensity therapy at step 3 of IAPT were assessed.

Setting

Recruitment took place within one IAPT mental healthcare service of a National Health Service Trust in England over a 9-month period, between September 2016 and June 2017. Like most other IAPT providers, the service experiences high levels of demand on their service and, consequently, they struggle with waiting lists due to a shortage of trained professionals who can provide HIT. HIT service provision at the site has undergone a transformation in its delivery of evidence-based treatments. The service introduced a therapeutic package for those requiring HIT. Service users were offered iCBT before commencing face-to-face treatment. Clients were monitored throughout the intervention, and any deterioration in their symptoms was responded to. Alternatively, if an appointment for face-to-face therapy (high-intensity therapy) became available, they were offered this treatment to begin with. Options available at step 3 for escalation include individual CBT (face-to-face and some online counseling), face-to-face delivered primary care counseling, or interpersonal therapy.

Participants

Our participants included both clinicians who worked at the IAPT service and clients who were referred to the service and for whom HIT was indicated as suitable. Clients who were deemed suitable for a typical HIT intervention and subsequently selected iCBT as part of their therapeutic package were invited to the study by their clinician. Criteria for receiving HIT in IAPT consists of clients with more severe presentations of anxiety and/or depression, client presentations that did not subside with a low intensity (step 2) treatment, or having a disorder that is not typically seen at the low intensity level (e.g. social anxiety). As with all IAPT services, substance abuse that is actively contributing to symptoms is an exclusion criterion, and these clients are referred on to specialist services for treatment.

All clinicians and PWPs supporting clients on SilverCloud as a prequel to HIT were eligible to participate in the study. Therefore, clinician participants consisted of a combination of clinical psychologists, counseling psychologists, and PWPs employed by the service.

Procedure

Client Procedure

Eligible participants were informed of the study through their clinician and invited to take part during their assessment appointment. On sign-up, participants were presented with information sheets for the study, and were also invited to discuss their participation with their clinician. Those willing to participate were then required to digitally sign to give their informed consent for participation. Those who declined consent or decided to withdraw from the study upon commencing treatment were requested to contact their clinician at the healthcare service, who would re-assess the client and assign them to SilverCloud treatment-as-normal or another intervention before their face-to-face appointment became available. Once participants finished their course of SilverCloud treatment or the waiting period came to an end, they progressed to either group therapy, face-to-face counseling, face-to-face CBT, or CBT delivered by a clinician via the internet.

Throughout their use of services, participants were asked to complete the minimum data set, as per the national requirements regarding IAPT services. In addition, they were asked to complete the Scale to Assess the Therapeutic Relationship—Patient Version (STAR-P), a measure of therapeutic alliance from the client’s perspective, during the 8-week supported period of the SilverCloud intervention.

Clinician Procedure

Clinicians were presented with a notification on their user accounts of the iCBT intervention that alerted them to the opportunity to participate in the research study. After reading the information sheet, they gave their consent to participate through their digital signature. Clinicians using the SilverCloud dashboard were able to monitor their client’s progress throughout the 8-week supported period of the intervention, and they regularly gave them feedback and responded to the work they had completed. Following each of these iCBT review sessions, clinicians were invited to complete the Scale to Assess the Therapeutic Relationship—Clinician Version (STAR-C) through their user accounts on the platform, which assesses the therapeutic relationship from the clinician’s perspective. Clinicians were also invited to participate in a qualitative semi-structured interview pertaining to acceptability of iCBT as a prequel to HIT.

Risk Management

At initial assessment and throughout treatment, clients were assessed for risk in line with routine clinical practice. The initial assessment for entry into services included questions of whether clients could maintain their safety while on the waiting list. Those who exceeded the cutoff score for risk in terms of self-harm on the screening questions were not eligible to participate in the study and were referred for additional support. Integrated risk measures in the SilverCloud platform allowed for the monitoring of any changes in risk for clients throughout the program. For example, if the client scored above 0 on the self-harm item of the PHQ-9, an alert would be sent through to their clinician, who could then escalate it appropriately within the established clinical governance structure. It is important to note that SilverCloud was not presented to clients as a program capable of providing crisis support, and this was further emphasized through informed consent, the client information sheet, and the user contract. Significant adverse events (SAEs) were handled in-service by the clinical team and were escalated appropriately.

Completers vs. Dropout

Treatment dropout was defined using the IAPT Care Spell End Code (40) which collects the reason for service exit as determined by the clinician. In this study, clients were categorized into two categories: completers comprising the service exit reasons “completed scheduled treatment” and “referred to other service” and dropouts comprising “dropped out of treatment (unscheduled discontinuation).” Other service exit reasons were not observed in this study population. As 10 clients were still in treatment by the study end point, their dropout status was classed as missing data.

Medication Status

Medication status was defined using the patient’s medication status at assessment as recorded in the patient management system. The options could be: “prescribed and taking,” “prescribed and not taking,” and “not prescribed.” Prescribed and not taking and not prescribed were combined within the same group (no medication), since both groups were not taking medication.

Intervention

SilverCloud delivers CBT-based online interventions for anxiety disorders, depression, and also comorbid depression and anxiety. Each program is compliant with National Institute for Health and Care Excellence (NICE) guidelines for the use of CBT in treatment, and is composed of eight modules that follow evidence-based CBT principles. They include tools such as self-monitoring and thought recording, behavioral activation, cognitive restructuring, and challenging core beliefs; all of which are central to the learning goals of the program. Research to date on the SilverCloud interventions has yielded significant positive clinical outcomes (41, 42).

Within the IAPT program, SilverCloud is typically delivered as a stand-alone low-intensity (step 2) intervention, or as an adjunct to high-intensity therapy. Specifically, for this study, participants were signed up at point of assessment and received reviews every 10–14 days via the platform. Once a participant is registered and assigned to a program on the iCBT platform, they receive a message from their clinician at their first login. This message welcomes them to the program, highlights its numerous aspects, and encourages them in the use of the program. Every fortnight, a clinician logs on and review participants’ progress, leaving feedback for them and responding to the work they have completed.

Data Collection

The following data were collected (Table 1):

TABLE 1

Table 1 Study measures and assessment times.

Participant Self-Reported Outcomes

Routinely Collected Data (The IAPT Minimum Data Set). This is a battery of psychometric measures common to all IAPT services collected as part of treatment-as-usual in all services. This battery primarily consists of the Patient Health Questionnaire-9 (PHQ-9), the Generalized Anxiety Disorder-7 (GAD-7), and the Work and Social Adjustment (WSAS). These questionnaires are administered to participants in all interventions at specific points in the patient pathway, for example at assessment, during treatment, and at post-treatment.

Patient Health Questionnaire-9 (43, 44) is a self-report measure of depression that has been widely used in screening, in primary care, and for research. The PHQ-9 items reflect the diagnostic criteria for depression outlined by the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition—Text Revision (DSM-IV-TR) (45). Summary scores range from 0 to 27, where larger scores reflect a greater severity of depressive symptoms. Cutoff scores for the PHQ-9 include none: 0–4, mild: 5–9, moderate: 10–14, moderately severe: 15–19, and severe: 20–27. The PHQ-9 has been found to discriminate well between depressed and non-depressed individuals using the clinical cutoff of total score ≥10, with good sensitivity (88.0%), specificity (88.0%), and reliability (.89) (43, 44).

Generalized Anxiety Disorder-7 (46); GAD-7 comprises seven items measuring symptoms and severity of GAD based on the DSM-IV diagnostic criteria for GAD. Scores on the GAD-7 range from 0 to 21, where a higher score reflects greater severity of anxiety symptoms. The GAD-7 has good internal consistency (α = .92) and good convergent validity with other anxiety scales (46). Higher scores indicate greater severity of symptoms. Cutoff scores for the GAD-7 include minimal: 0–4, mild: 5–9, moderate: 10–14, and severe: 15–21. The GAD-7 has increasingly been used in large-scale studies as a generic measure of change in anxiety symptomatology, using a cutoff score of 8 (47–49).

Work and Social Adjustment Scale is a simple, reliable (α > .75), and valid measure of impaired functioning (50). The measure contains five self-report items that look at how the disorder impairs the client’s ability to function day to day on five dimensions: work, social life, home life, private life, and close relationships. A score of 0–9 on the measure indicates subclinical symptoms, 10–19 is indicative of significant functional impairment but lower clinical symptoms, and above 20 suggests both severe clinical symptoms and functional impairment (41).

Scale to Assess the Therapeutic Relationship—Patient Version (51) is the client version of the scale to assess the therapeutic relationship in community mental healthcare. It was developed to be used in adult patients with mental health problems in psychiatric care. It is composed of 12 items and three subscales. The first subscale, positive collaboration, explores the general quality of the therapeutic relationship between patient and clinician. The second subscale, positive clinician input, illustrates the extent to which the client perceives their clinician positively regarding them. The third subscale, non-supportive clinician input, illustrate clients’ perceived problems within the therapeutic relationship. The range of scores for the STAR-P is 0 to 48, with a higher score suggesting a better therapeutic relationship. The measure has shown excellent psychometric properties.

Access Data

Several types of treatment access data were collected as part of this research project. These data contained the following variables:

● Waiting time between initial client triage and access to iCBT (in days)

● Time spent by each client in iCBT (in days)

● Waiting time between initial client triage and access to HIT (in days)

● Time spent by each client in face-to-face HIT (in days)

● Number of iCBT reviews received by each client

● Number of HIT sessions attended by each client

Clinician-Reported Outcomes

Scale to Assess the Therapeutic Relationship—Clinician Version (51) is the clinician version of the scale to assess therapeutic relationship in routine scale. It was developed to be used in adult patients with mental health problems in psychiatric care. Similar to the client counterpart (STAR-P), the measure is composed of 12 items and three subscales. Firstly, positive collaboration, which explores the general quality of the relationship and the overall degree to which the relationship works. Secondly, positive clinician input measures to what extent clinicians encourage, understand, and support the patient. Lastly, emotional difficulties assesses the clinician’s feeling that they cannot empathize with and are not accepted by the patient. The range of scores for the STAR is 0 to 48, with a higher score suggesting a better therapeutic relationship. The measure has shown excellent psychometric properties.

Semi-structured interview. The semi-structured interview schedule for clinicians was developed ad hoc for the present study. It aimed to explore clinicians’ perceptions regarding the content of the program, their experience offering the reviews, and the clinical utility of the intervention being offered during the waiting period and before accessing HIT.

Ethics

This study was carried out in accordance with the recommendations of the United Kingdom’s Research Ethics Service (52) with written informed consent from all participants (see section 2.4.1, Client Procedure, for procedure). All participants gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by Wales Rec 7 (reference number: 16/WA/0257).

For participants that opted out of the SilverCloud intervention or the research, it is important to note that they were made aware that their place on the waiting list for services would not be jeopardized by their non-participation. Furthermore, for those who chose to partake in the intervention, their choice did not prolong their stay on the waiting list or their access to HIT.

Statistical Analyses

Estimates of caseness, reliable change, and recovery were calculated using IAPT recovery criteria. A client is at “caseness” if they score above the clinical caseness threshold on the PHQ-9 (≥ 10) or GAD-7 (≥ 8). A client is classed as recovered if they move from caseness at baseline to below the clinical caseness threshold post-treatment. Reliable improvement is defined as a decrease in either or both measures greater than or equal to the reliable change index (PHQ-9 RCI = 6, GAD-7 RCI = 4) with no parallel increase in score on either measure greater than or equal to the RCI. Clients that both recover and show a reliable improvement in their score(s) are said to be in reliable recovery.

Treatment effects on primary outcome measures from baseline to iCBT treatment exit to service exit were assessed using linear mixed models fit with restricted maximum likelihood (REML) in the R (53) package lme4 v1.1-13 (54). This analysis was conducted separately for each measure, and missing data were assumed to be missing at random (MAR). Dropout status, defined according to the IAPT service exit reason as detailed in Materials and Methods, was included as a factor in each model. All participants attended part or all of the scheduled iCBT intervention. However, many participants did not thereafter attend any high-intensity treatment. To investigate whether this factor was associated with more positive or negative outcomes, clients were classified according to whether they attended at least one high-intensity treatment appointment (High Intensity Yes) or no high-intensity treatment appointments (High Intensity No). Medication status was also included in the model, where participants who were prescribed and not taking and those who were not prescribed were combined within the same group (medstatus No), and the other group referred to people who were prescribed and taking medication (medstatus Yes). Model selection was carried out by first creating a full model including all fixed factors (time point, dropout status, medication status, high-intensity attendance, and all interactions between them) and then performing backward elimination of effects from this full model. Backward elimination and model evaluation were completed using the step function from R package lmerTest v3.0-0 (55), which determined the optimum model. Post-hoc comparisons of least-square mean predictions from the optimum models were carried out using the R package lsmeans v2.27 (56).

Progression of the therapeutic relationship throughout iCBT treatment was modeled separately from the client’s perspective (with STAR-P scores) and clinician’s perspective (with STAR-C scores). For each measure, scores were modeled using a linear mixed model with fixed factors of time (a continuous variable: number of days after iCBT treatment start date), dropout status, and the interaction between them and client as a random factor. Post-hoc comparisons of least-square mean predictions at the predicted treatment exit day were carried out using the R package lsmeans v2.27.

Lastly, due to the low response rate from clinicians for the qualitative interviews, a descriptive–interpretative qualitative approach was used (57). This method was used to identify clinicians’ viewpoints regarding acceptability of the iCBT intervention as a prequel to step 3 services. Acceptability was defined as the degree to which the intervention, from the clinician’s viewpoint, was acceptable in terms of content, relevancy, and ability to engage the user in a therapeutic relationship. For the analysis of the interviews, these were transcribed, and thereafter, meaning units were extracted from the responses of the clinicians. Meaning units are parts of data that even standing out of context provide a piece of meaning to the reader. These meaning units were clustered together to form categories around acceptability. A more detailed description of these categories, including quotes that mirror each of them, is depicted in Table 2.

TABLE 2

Table 2 Definition of the categories used in the qualitative analyses of Clinician views on acceptability.

Results

Overview

One hundred and twenty-four (N = 124) clients were recruited to the study (Figure 1). One client did not engage after baseline, did not complete any post-assessment questionnaires, and therefore could not be included in analyses, leaving an effective sample size of N = 123. Sociodemographic characteristics of the population at baseline are presented in Table 3. Baseline scores were unavailable for 13 clients as the authors were unable to guarantee the integrity of their data at this time point due to a lack of clarity around previous treatment received and instances of a double assessment. Service exit scores were unavailable for 10 clients who were still in treatment by the end of the study. All 123 clients were included in analyses of primary outcome measures where models were fit with REML. However, the 23 clients with incomplete data could not be included in estimation of reliable change and recovery rates. All 123 clients were included in analyses of therapeutic alliance. Eleven clinicians were recruited and participated in the study.

FIGURE 1

Figure 1 Trial flowchart, including numbers included in each statistical analysis.

TABLE 3

Table 3 Sociodemographic characteristics of clients at baseline.

Treatment Adherence: Completers and Dropouts

One hundred and thirteen (n = 113) clients’ IAPT treatment episode had terminated by the study end point—68 completers and 45 dropouts as defined by their IAPT Care Spell End Code explained in the Materials and Methods. Ten clients were still in treatment by the study end point, so their dropout status was classed as missing data.

Completers attended on average 1.2 more iCBT appointments than dropouts [completers mean = 3.8, 95% CI (3.3, 4.3); dropouts mean = 2.6, 95% CI (2.1, 3.1); t (108) = 3.5, p < .001). There was no significant difference in the number of unattended iCBT appointments between completers and dropouts [completers mean = 1.9, 95% CI (1.5, 2.3); dropouts mean = 2.4, 95% CI (2.0, 2.7); t (109) = −1.9, p = .067]. There was no difference in the average amount of time in days spent in iCBT treatment [mean = 46.8, 95% CI (43.8, 49.7); t (91) = 1.3, p = .219].

Completers also attended on average 8.1 more high-intensity treatment appointments than dropouts [completers mean = 9.9, 95% CI (8.5, 11.2); dropouts mean = 1.8, 95% CI (0.6, 3.0); t (110) = 9.0, p < .001]. There was no significant difference in the number of unattended high-intensity treatment appointments between completers and dropouts [completers mean = 2.5, 95% CI (1.7, 3.2); dropouts mean = 2.1, 95% CI (1.4, 2.7); t (111) = 0.8, p = .437]. On average, completers spent almost twice the amount of days in high-intensity treatment than dropouts [completers mean = 147.2, 95% CI (122.9, 171.5); dropouts mean = 74.5, 95% CI (41.4, 107.6); t (63) = 3.6, p < .001].

Reliable Change, Recovery, and Reliable Recovery

One hundred (n = 100) clients had a full set of questionnaire scores from all time points (baseline, iCBT exit, and service exit) and could be included in estimation of reliable improvement rates. Fifty-eight percent (58%; n = 58) exhibited reliable improvement from baseline to iCBT exit, and 70% (70 clients) exhibited reliable improvement from baseline to service exit.

Ninety-nine (n = 99) clients were above the clinical caseness threshold at baseline and could be included in estimation of recovery and reliable recovery rates. Recovery and reliable recovery rates for these 99 clients are detailed in Table 4. Twenty-two (n = 22) clients had achieved recovery by the time of iCBT exit and 20, of these had reliably recovered. Thirty-three (n = 33) clients were in recovery at the point of service exit, all of which had reliably recovered. The recovery rate was significantly higher for those 56 clients who completed treatment (46%) compared to the 43 clients who did not complete treatment (16%) (X-squared = 10.7, df = 2, p = .005).

TABLE 4

Table 4 Recovery and reliable recovery rates for completers and dropouts.

Primary Outcome Treatment Effects

For all three primary outcome measures, the optimum linear mixed model included only time point, dropout status, and the interaction of time point and dropout status as fixed effects and client as a random effect. High-intensity treatment attendance did not add significantly to any of the models indicating that dropout is a more accurate predictor of treatment outcome for all three measures. Medication status was found to be a significant variable in the model for PHQ-9 and not for GAD-7 and WSAS. Post-hoc comparisons within the PHQ-9 model did not find any interaction with any of the other variables and, therefore, was discarded for being included in further analyses (Supplementary Table S1).

Post-hoc analyses of the optimum linear mixed models showed a significant reduction of severity scores from baseline to iCBT exit and again from iCBT exit to service exit for all three primary outcome measures (Table 5). For all measures, a large reduction in severity score took place in the period from baseline to iCBT exit for both completers and dropouts. A further reduction was observed from iCBT exit to service exit only for completers. Dropouts showed no significant change.

TABLE 5

Table 5 Differences in least-square mean questionnaire scores between time points.

For the PHQ-9, scores were reduced by an average of 3.6 points from baseline to iCBT exit [Cohen’s d = 0.61, 95% CI (0.34, 0.88)], with no significant difference between completers and dropouts at either the baseline or iCBT exit time point. Completers had a further reduction of on average 4.0 points from iCBT exit to service exit [Cohen’s d = 0.64, 95% CI (0.29, 0.99)], whereas those who disengaged with the service showed no further change in score at service exit [Cohen’s d = −0.05, CI (−0.47, 0.37)].

Similarly, baseline GAD-7 scores were reduced by on average 3.2 points at iCBT exit time point [Cohen’s d = 0.69, 95% CI (0.42, 0.97)] and a further 4.0 points at service exit for completers [Cohen’s d = 0.83, 95% CI (0.47, 1.18)]. Dropouts did not show this further reduction in scores at service exit [Cohen’s d = 0.03, 95% CI (−0.39, 0.45)].

Change in WSAS score, which estimates severity of work and social adjustment impairment, followed a similar pattern with a substantial reduction (mean = 2.4 points) in severity score at iCBT exit time point for the whole population [Cohen’s d = 0.31, 95% CI (0.04, 0.58)]. Completers showed a further reduction of 4.1 points [Cohen’s d = 0.60, 95% CI (0.25, 0.94)], whereas dropouts did not have any significant change in score from iCBT exit to service exit [Cohen’s d = 0.15, 95% CI (−0.27, 0.57)].

Therapeutic Alliance—Client’s Perspective

In total, 134 STAR-P questionnaires were collected, representing the therapeutic relationship of 56 clients from the client’s perspective. The total amount of time in days spent in iCBT treatment was similar for completers (mean = 47.279, SD = 15.757) and dropouts (mean = 46.145, SD = 17.274). No significant change in average STAR-P scores were seen for dropouts. For completers, a significant increase in STAR-P scores of on average 3.9 points was observed from baseline (day 0) to average end of treatment (day 46) (Table 6).

TABLE 6

Table 6 Difference in least-square mean STAR-P total scores and sub-scores from baseline to iCBT treatment exit for completers and dropouts.

Therapeutic Alliance—Clinician’s Perspective

In total, 371 STAR-C questionnaires were collected, representing the therapeutic relationship of 82 clients from their clinician’s perspective. For completers, no significant change in STAR-C scores was observed from baseline (day 0) to end of treatment (day 47). For dropout clients, the STAR-C scores declined significantly by on average 5.4 points from baseline to end of treatment (day 46) (Table 7).

TABLE 7

Table 7 Difference in STAR-C total scores and sub-scores from baseline to iCBT treatment exit for completers and dropouts.

Practitioner Feedback

Of the 11 clinicians, 2 consented to a follow-up interview to further explore their experiences about acceptability. One male CBT therapist and one female PWP constituted the interview sample. Table 2 describes each of the four identified categories and supporting quotes from both participants.

Discussion

The current study examined an innovative model of service delivery, integrating digital interventions as a frontline intervention before accessing HIT. On average, clients were on the moderate-severe range for depressive symptoms and on the severe range for anxiety symptoms at baseline. Regarding treatment outcomes, there was a decrease in symptoms upon completion of both iCBT and HIT interventions with effect sizes ranging from moderate to large, and around 20% of the sample achieved reliable recovery in advance of starting face-to-face therapy. These results corroborate previous results demonstrating the benefits of iCBT in reducing symptoms of depression and anxiety in more severe presentations of depression and anxiety (22). The results also indicate that iCBT can be beneficial as a prequel to high-intensity therapy in IAPT and in doing so facilitate clients to gain access to a frontline evidence-based intervention while waiting for face-to-face therapy, which is in line with the findings of other studies in blended interventions where internet interventions where offered before face-to-face therapy.

Despite the majority of clients transitioning from severe to mild presentations by point of service exit, some patients were still at caseness and, by definition, not recovered at this point. One potential explanation for these results is that, although current iCBT treatments are capable of producing improvements in symptoms, they may not be sufficient for producing lasting effects in severely anxious and depressed clients. Further treatments or booster sessions could therefore be necessary to achieve higher recovery rates (58, 59). This was especially noticeable in the dropout group, whose numbers achieved significantly less recovery rates at point of service exit. Completers received an adequate amount of HIT sessions to alleviate their symptoms (60), whereas those who dropped out completed only two HIT sessions on average and therefore any improvements made during iCBT were not further enhanced due to dropout from HIT. Understanding exactly what works for whom when integrating digital interventions into care pathways is an area of growing knowledge that would help to understand, and predict, future dropouts (61).

Clients spent on average 47 days in iCBT treatment and had 3.4 reviews on average during this period, meaning one review every 10 to 12 days. The observed administration of the intervention was below the intended use, that is, to be administered over a period of 8 weeks and for clients to receive six supporter reviews during this time (22, 62). This finding illustrates the importance of investigating the implementation of iCBT in novel contexts, since the ideal standard of intervention delivery was not adhered to in this study, but still produced clinical benefits. Further research regarding implementation of these interventions can illustrate and account for the changes that occur when interventions are implemented in novel contexts, as well as the barriers to implementation that are encountered (63). The value of the iCBT intervention is further reflected in the long waiting times clients experienced to accessing treatment. Average wait time was 158 days, and implementing iCBT can provide clients in these pathways with a frontline intervention in contexts where resources are scarce. In this instance specifically, waiting times for clients were reduced by 30%.

The dropout rate observed in this study (37%) is lower than the 45% rate of individuals not ending their course of therapy reported by IAPT (64). Reasons for dropout were not collected, but one study in this context found that waiting for an excessive period and the lack of contact during the wait lead people to dropout from services (65). In the current study, the long waiting list could have influenced people to disengage from services, and it might also have had an effect on improvement rates, as they have been shown to play a detrimental effect on outcomes (66). In regards to the wider literature on blended care interventions, a review conducted by Erbe et al. (67), found some studies where iCBT was also offered as a prequel to face-to-face therapy. The studies included in this review found dropout rates in the ranges of 30% to 59% across their samples (68–70), which is in line with what we observed in this study.

Therapeutic alliance from the client perspective increased significantly from pre to post for those who completed the scheduled intervention, while no change was observed in the case of those who dropped out before finishing their scheduled sessions. The positive alliance outcomes for completers showed that a strong therapeutic alliance can be established and maintained online (30, 31). These results further align with client perceptions of the role of support in iCBT, where different studies have shown that clients found it helpful and motivating during the intervention period (71, 72). The lack of change in perception of alliance for dropouts could be explained by the group receiving significantly less iCBT reviews and not getting enough contact to perceive an increase in alliance, which may take more time to develop in iCBT (73).

Therapeutic alliance from the clinician perspective showed the opposite pattern. Clinicians perceived no change in therapeutic alliance with clients who remained in treatment and perceived a significant decrease in alliance with those who dropped out from the services. The maintenance of alliance ratings perceived by the therapists over time on completers is similar to another study where these ratings were perceived as high, but no significant change was observed from pre- to post-treatment (34). The decrease in alliance with clients who dropped out could be explained by the fact that these clinicians conduct reviews for many clients in one scheduled review period and that they mostly base their perception of alliance on the usage of the platform while receiving little direct feedback from their client. In this sense, if they noticed some clients were engaging less with the platform, they may potentially feel a decrease in their perceptions of alliance over the course of the iCBT treatment. This interpretation is supported by the qualitative interviews, where clinicians said they put less effort in the reviews (i.e. write less and use more standard responses) if clients were not engaging, which could influence the levels of therapeutic alliance. Furthermore, these results could also be reflecting clinician bias toward relationship formation online, a result observed in other studies where clinicians also found iCBT as impersonal and thought that it was not feasible to create a therapeutic relationship in iCBT (74). However, more studies are needed to explore therapists’ perceptions of alliance in iCBT, and specially within regular clinical settings, in order to determine if these patterns are also observed in different settings and countries.

Although we gathered only a small amount of feedback from a small number of clinicians and therefore caution is advised in interpreting the comments, some interesting points arise. For starters, a positive aspect of the review process highlighted by clinicians was the asynchronous nature of the clinician–client contact, which allowed for them to reflect and provide more insightful feedback to their client. However, they reported that once they sent the message, there was no way for them to know how the client would interpret it. This uncertainty was further emphasized by the perceived removed nature of the therapeutic relationship, which they believed had a negative impact and made it difficult to establish a therapeutic alliance. Both clinicians agreed that the intervention would work best for clients who are self-motivated to work through the content with minimal guidance. These reports are in line with findings about the importance of the readiness to engage in therapy and the sense of self-directedness for iCBT (74), which have been found to be predictors of adherence (75). They also agreed that for step 3 service provision, an iCBT intervention should not be offered as a stand-alone solution but as an adjunct (prequel) to face-to-face therapy. This finding is also in line with the feedback obtained from therapists in other studies, who consider iCBT as an adjunct to face-to-face therapy more than a replacement and are more open to the possibility of offering blended interventions (74, 76, 77).

Implications for Psychological Therapy Services

Some studies have explored the effects of internet and computerized CBT interventions as an adjunct to face-to-face therapy or as a treatment alternative in similar settings, showing mixed results. Thus, some studies have explored the beneficial effects of iCBT interventions when deployed at specialized care settings (78, 79), whereas others have not found any value in adding iCBT as an adjunct to face-to-face therapy (69). These inconsistencies in the findings may well be attributed to differences in implementation strategies and contextual factors, such as training with supporters, clinician’s attitudes toward the interventions, and level of integration of iCBT interventions into services and treatment pathways (80). Future studies should explore and consider the impact of different implementation factors on the uptake and effects of these interventions.

The findings observed in this study also have some implications for IAPT, where the national target for recovery is 50%, but there is no extant benchmark to untangle the recovery rates across different steps in IAPT. For the total sample, clients who received iCBT as a frontline intervention achieved a 22.2% recovery rate upon iCBT completion. Thereafter, with face-to-face therapy, this rate of recovery rose to 33.3%. Of importance, those who completed their entire course of treatment achieved a 46% recovery rate. These results provide initial supporting evidence for the use of iCBT as an adjunct to face-to-face therapies for more severe presentations. They do, however, stand in contrast to current guidance and practice in IAPT (12, 13) for the treatment of depression and anxiety, where digital interventions are recommended for clients with mild-moderate symptoms. As a consequence, these guidelines can also limit the scope and applicability of iCBT interventions to other contexts, such as high-intensity services for clients with more complex needs. Building on the work of prior research in the field (7, 21, 22) and the results from the current study, the authors acknowledge a need for further controlled research to robustly articulate the impacts of digital interventions in order to transform high-intensity service delivery. This will contribute to the development of clinical guidelines that incorporate digital interventions alongside current modalities of treatment for individuals with more severe presentations of depression and anxiety.

Limitations

Naturalistic observation studies, such as this one, have some limitations due to their uncontrolled nature. In future studies, the magnitude of the effect produced by the intervention could be more clearly characterized through comparison with a control group with no access to the iCBT intervention. Primary outcomes were not collected before starting high-intensity therapy, and there was a gap between iCBT exit and the start of high-intensity therapy, so the true severity of symptoms when starting high-intensity therapy was unknown. Information about when the clients dropped out from services was unclear, making it difficult to establish whether effects could be attributed to iCBT or the HIT therapy. The current study recorded no SAEs for any participant, but it is important for the field of internet interventions to explore adverse events as they occur within natural service. For example, dropout from service is not rigorously followed up within IAPT, which limited the researchers in expanding further on those in the "dropout" group. In regards to the interviews with clinicians, the low response rate might have biased the feedback, where perhaps only those with a positive attitude toward digital interventions accepted to participate. The low response rate is likely to be due to the busy schedule the clinicians work under; however, future studies should confirm this hypothesis. Finally, date of referral was not collected, so we cannot ensure how long clients waited to access the iCBT program from referral. However, from IAPT reports and our experience with different services, the delay between referral and access to iCBT use to be minimal, and this was also probably the case in this study.

Conclusion

The current research has illustrated the potential effectiveness and benefit of implementing an iCBT intervention as a prequel to face-to-face therapy in individuals with severe presentations of depression and anxiety in naturalistic settings. As indicated by Andersson and colleagues (81), open studies with no control group play a key role in clinical effectiveness research and, as such, the present paper constitutes a relevant contribution to the advancement of the field of iCBT. The results showed that iCBT was a valuable option regarding waiting time reduction and clinical efficiency. As stated by the clinicians participating in this research, communicating the foundations of CBT through an internet-delivered intervention can be of high value to a service regarding time, cost, and clinical efficiency. Although generalizability of the findings is limited by the uncontrolled nature of the research, future investigations into this area are warranted to further validate the potential benefits of incorporating iCBT as a frontline intervention to high-intensity services. The authors also acknowledge the need for research to be conducted on the implementation of these types of interventions into these contexts, using evidence-based methodologies, in order to determine how iCBT interventions can be best implemented into client pathways while accounting for levels of clinical risk. If implemented correctly within step 3 pathways, the clinical and cost benefits observed within mild-moderate clinical ranges and step 2 (low-intensity) services in IAPT could be successfully replicated and illustrated.

Knowledge Contribution

Depression and anxiety disorders are a leading cause of disability worldwide. Internet-delivered cognitive behavior therapy (iCBT) has been successfully used to treat mild-moderate depression and anxiety disorders, but its applicability to more severe presentations is relatively undocumented. Typically severe presentations of depression and anxiety are treated using high-intensity interventions (HIT) such as face-to-face therapy. This study explored the potential benefits of iCBT as a prequel to face-to-face therapy for service users with depression/anxiety in a high-intensity therapy pathway for more severe presentations of depression and anxiety. Results from this preliminary research are promising; positive changes in clinical outcomes were observed for those completing iCBT, which further improved post-HIT. The relationship established between clinicians and patients was important and influenced by patient dropout. The results illustrate the potential utility of iCBT when used as a prequel to HIT. The research also presents a novel knowledge contribution to iCBT and the wider field of psychological therapies and paves the way for future investigations.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This study was carried out in accordance with the recommendations of the United Kingdom’s Research Ethics Service with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by Wales Rec 7 [reference number: 16/WA/0257]. For participants that opted out of the SilverCloud intervention or the research, it is important to note that they were made aware that their place on the waiting list for services would not be jeopardised by their non-participation. Furthermore, for those who chose to partake in the intervention, their choice did not prolong their stay on the waiting list or their access to step 3 interventions.

Author Contributions

DD and DR conceptualized the initial design of the study. AE led on the development of the first and subsequent drafts of the manuscript, with significant contributions from both DD and SC. DD, AE, and SC designed the data analytic plan, and SC implemented it. CC conducted and analyzed the qualitative interviews. DR reviewed the manuscript and provided feedback for each draft.

Conflict of Interest

DD, AE, SC, CC, and DR are all employees of SilverCloud Health. DD, AE, and DR are also researchers with the E-Mental Health Research Group of Trinity College, Dublin, Ireland. This research was funded by SilverCloud, the commercial company marketing the iCBT system studied here.

As principal funder for this research, SilverCloud Health provided the necessary research resource to design the study, collect, and analyze the data and write the report. The researchers adhere to the ethical standards of research and reporting of the National Health Service (NHS) and Trinity College Dublin (https://www.tcd.ie/research/dean/ethics/).

Acknowledgments

Thanks to all of the participants who contributed to this research study. Thanks to the host NHS Trust and their Research & Development department for hosting this study. Thanks to Colette Clyne for assisting with qualitative data collection.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2019.00902/full#supplementary-material

References

1. Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study (2010). Lancet (2013). 382(9904):1575–86. doi: 10.1016/S0140-6736(13)61611-6

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Nathan PE, Gorman JM. A guide to treatments that work. 4th ed. Nathan PE, Gorman JM, editors. Oxford University Press: New York, NY, US (2015). doi: 10.1093/med:psych/9780195304145.001.0001

CrossRef Full Text | Google Scholar

3. McHugh RK, Whitton SW, Peckham AD, Welge JA, Otto MW. Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: A meta-analytic review [Internet]. J Clin Psychiatry (2013) 74, 595–602. doi: 10.4088/JCP.12r07757

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Richards D, Enrique A, Palacios J. Internet-delivered Cognitive Behavioural Therapy. In: Parry S, editor. The Handbook of Brief Therapies: A Practical Guide. [Internet]. Sage, London (2019). p. 248. Available from: https://www.researchgate.net/publication/330761832_Internet-delivered_Cognitive_Behavioural_Therapy. doi: 10.5772/intechopen.71412

CrossRef Full Text | Google Scholar

5. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J Anxiety Disord (2018) 55:70–8. doi: 10.1016/j.janxdis.2018.01.001

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Wells MJ, Owen JJ, McCray LW, Bishop LB, Eells TD, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression in primary care. Prim. Care Companion CNS Disord (2018) 20(2). doi: 10.4088/PCC.17r02196

CrossRef Full Text | Google Scholar

7. Wright JH, Owen JJ, Richards D, Eells TD, Richardson T, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression: a systematic review and meta-analysis. J Clin Psychiatry (2019) 80(2). doi: 10.4088/JCP.18r12188

CrossRef Full Text | Google Scholar

8. Titov N, Dear B, Nielssen O, Staples L, Hadjistavropoulos H, Nugent M, et al. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv (2018) 13:108–15. doi: 10.1016/j.invent.2018.07.006

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: A meta-analytic review. Cognit Behav Ther (2009) 38(2):66–2. doi: 10.1080/16506070802694776

CrossRef Full Text | Google Scholar

10. Johansson R, Andersson G. Internet-based psychological treatments for depression. Expert Rev Neurother (2012) 12(7):861–70. doi: 10.1586/ern.12.63

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Palmqvist B, Carlbring P, Andersson G. Internet-delivered treatments with or without therapist input: does the therapist factor have implications for efficacy and cost? Expert Rev PharmacoEcon Outcomes Res (2007) 7(3):291–7. doi: 10.1586/14737167.7.3.291

PubMed Abstract | CrossRef Full Text | Google Scholar

12. National Institute for Health and Care Excellence. Computerised cognitive behaviour therapy for depression and anxiety. London, UK: NICE (2006).

Google Scholar

13. National Institute for Health and Care Excellence. The treatment and management of depression in adults. London, UK: NICE (2009).

Google Scholar

14. Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry (2014) 13(1):4–11. doi: 10.1002/wps.20083

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Andrews G, Williams AD. Up-scaling clinician assisted internet cognitive behavioural therapy (iCBT) for depression: a model for dissemination into primary care. Clin Psychol Rev (2015) 41:40–8. doi: 10.1016/j.cpr.2014.05.006

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Bower P, Gilbody S. Stepped care in psychological therapies: access, effectiveness and efficiency. Br J Psychiatry (2005) 186(01):11–7. doi: 10.1192/bjp.186.1.11

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Clark DM. Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. Int Rev Psychiatry (2011) 23(4):318–27. doi: 10.3109/09540261.2011.606803

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Richards D, Chellingsworth M, Hope R, Turpin G, Whyte M. National programme supervisor materials to support the delivery of training for psychological wellbeing practitioners delivering low intensity interventions. 1st Ed. London, UK: Rethink (2010).

Google Scholar

19. Richards D, Whyte M. Reach Out: National programme student materials to support the delivery of training for psychological wellbeing practitioners delivering low intensity interventions. 3rd Ed. London, UK: Rethink (2011).

Google Scholar

20. de Graaf LE, Gerhards SAH, Arntz A, Riper H, Metsemakers JFM, Evers SMAA, et al. Clinical effectiveness of online computerised cognitive–behavioural therapy without support for depression in primary care: randomised trial. Br J Psychiatry (2009) 195(01):73–80. doi: 10.1192/bjp.bp.108.054429

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Bower P, Kontopantelis E, Sutton A, Kendrick T, Richards DA, Gilbody S, et al. Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ (2013) 346(7899):1–11. doi: 10.1136/bmj.f540

CrossRef Full Text | Google Scholar

22. Richards D, Duffy D, Burke J, Anderson M, Connell S, Timulak L. Supported internet-delivered cognitive behavior treatment for adults with severe depressive symptoms: a secondary analysis. JMIR Ment Heal (2018) 5(4):e10204. doi: 10.2196/10204

CrossRef Full Text | Google Scholar

23. Meyer B, Bierbrodt J, Schröder J, Berger T, Beevers CG, Weiss M, et al. Effects of an Internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial. Internet Interv (2015) 2(1):48–59. doi: 10.1016/j.invent.2014.12.003

CrossRef Full Text | Google Scholar

24. Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychother Theory Res Pract (1979) 16(3):252–60. doi: 10.1037/h0085885

CrossRef Full Text | Google Scholar

25. Bordin ES. Theory and research on the therapeutic working alliance: new directions. In: Horvath AO, Greenberg LS, editors. The working alliance: Theory, research, and practice. Oxford: John Wiley & Sons (1994). p. 13–37. Available from: https://psycnet.apa.org/record/1994-97518-001.

Google Scholar

26. Horvath AO, Symonds BD. Relation between working alliance and outcome in psychotherapy: a meta-analysis. J Couns Psychol (1991) 38(2):139–49. doi: 10.1037//0022-0167.38.2.139

CrossRef Full Text | Google Scholar

27. Martin DJ, Garske JP, Katherine Davis M. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol (2000) 68(3):438–50. doi: 10.1037//0022-006X.68.3.438

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Del Re AC, Flückiger C, Horvath AO, Symonds D, Wampold BE. Therapist effects in the therapeutic alliance-outcome relationship: a restricted-maximum likelihood meta-analysis. Clin Psychol Rev Pergamon (2012) 32:642–9. doi: 10.1016/j.cpr.2012.07.002

CrossRef Full Text | Google Scholar

29. Johansson H, Eklund M. Patients’ opinion on what constitutes good psychiatric care. Scand J Caring Sci (2003). 17(4):339–46. doi: 10.1046/j.0283-9318.2003.00233.x

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Pihlaja S, Stenberg JH, Joutsenniemi K, Mehik H, Ritola V, Joffe G. Therapeutic alliance in guided internet therapy programs for depression and anxiety disorders – a systematic review. Internet Interv Elsevier (2018) 11:1–10. doi: 10.1016/j.invent.2017.11.005

CrossRef Full Text | Google Scholar

31. Berger T. The therapeutic alliance in internet interventions: a narrative review and suggestions for future research. Psychother Res (2017) 27(5):511–24. doi: 10.1080/10503307.2015.1119908

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Andersson G, Paxling B, Wiwe M, Vernmark K, Felix CB, Lundborg L, et al. Therapeutic alliance in guided internet-delivered cognitive behavioural treatment of depression, generalized anxiety disorder and social anxiety disorder. Behav Res Ther (2012) 50(9):544–50.

PubMed Abstract | Google Scholar

33. Ertelt TW, Crosby RD, Marino JM, Mitchell JE, Lancaster K, Crow SJ. Therapeutic factors affecting the cognitive behavioral treatment of bulimia nervosa via telemedicine versus face-to-face delivery. Int J Eat Disord (2011). 44(8):687–91. doi: 10.1002/eat.20874

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Knaevelsrud C, Maercker A. Internet-based treatment for PTSD reduces distress and facilitates the development of a strong therapeutic alliance: a randomized controlled clinical trial. BMC Psychiatry (2007) 7(1):13. doi: 10.1186/1471-244X-7-13

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Preschl B, Maercker A, Wagner B. The working alliance in a randomized controlled trial comparing online with face-to-face cognitive-behavioral therapy for depression. BMC Psychiatry (2011) 11(1):189. doi: 10.1186/1471-244X-11-189

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Nordgren LB, Carlbring P, Linna E, Andersson G. Role of the working alliance on treatment outcome in tailored internet-based cognitive behavioural therapy for anxiety disorders: randomized controlled pilot trial. J Med Internet Res (2013) 15(1):e4. doi: 10.2196/resprot.2292

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Richards D, Duffy D, Burke J, Anderson M, Connell S, Timulak L. Supported internet-delivered cognitive behavior treatment for adults with severe depressive symptoms: a secondary analysis. JMIR Ment Heal (2018) 5(4):e10204. doi: 10.2196/10204

CrossRef Full Text | Google Scholar

38. Richards D, Timulak L, O’Brien E, Hayes C, Vigano N, Sharry J, et al. A randomized controlled trial of an internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression. Behav Res Ther (2015) 75:20–31. doi: 10.1016/j.brat.2015.10.005

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Richards D, Murphy T, Viganó N, Timulak L, Doherty G, Sharry J, et al. Acceptability, satisfaction and perceived efficacy of “Space from Depression” an internet-delivered treatment for depression. Internet Interv (2016) 5:12–22. doi: 10.1016/j.invent.2016.06.007

PubMed Abstract | CrossRef Full Text | Google Scholar

40. National Health Service. Improving Access to Psychological Therapies Care Spell End Code. In: NHS Data Model and Dictionary version 3 (2019). Available from: https://www.datadictionary.nhs.uk/data_dictionary/attributes/i/improving_access_to_psychological_therapies_care_spell_end_code_de.asp?shownav=1.

Google Scholar

41. Richards D, Timulak L, O’Brien E, Hayes C, Vigano N, Sharry J, et al. A randomized controlled trial of an internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression. Behav Res Ther (2015) 75:20–31. doi: 10.1016/j.brat.2015.10.005

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Palacios JE, Richards D, Palmer R, Coudray C, Hofmann SG, Palmieri PA, et al. Supported internet-delivered cognitive behavioral therapy programs for depression, anxiety, and stress in university students: open, non-randomised trial of acceptability, effectiveness, and satisfaction. JMIR Ment Heal (2018) 5(4):e11467. doi: 10.2196/11467

CrossRef Full Text | Google Scholar

43. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med (2001) 16(9):606–13. doi: 10.1046/j.1525-1497.2001.016009606.x

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Spitzer RL, Kroenke K, Williams JBW. Validation and utility of a self-report version of PRIME-MD: the PHQ Primary Care Study. J Am Med Assoc (1999) 282(18):1737–44. doi: 10.1001/jama.282.18.1737

CrossRef Full Text | Google Scholar

45. American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV-TR. Washington, DC: American Psychiatric Association (2000).

Google Scholar

46. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder. Arch Intern Med (2006) 166(10):1092. doi: 10.1001/archinte.166.10.1092

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Richards DA, Suckling R. Improving access to psychological therapies: phase IV prospective cohort study. Br J Clin Psychol (2009) 48(4):377–96. doi: 10.1348/014466509X405178

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Richards D, Duffy D, Blackburn B, Earley C, Enrique A, Palacios J, et al. Digital IAPT: the effectiveness & cost-effectiveness of internet-delivered interventions for depression and anxiety disorders in the Improving Access to Psychological Therapies programme: study protocol for a randomised control trial. BMC Psychiatry (2018) 18(1):59. doi: 10.1186/s12888-018-1639-5

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic Internet treatment for anxiety disorders: a randomized controlled trial. (2010) [cited 2019 Nov 8]; Available from: www.virtualclinic.org.au

Google Scholar

50. Mundt JC, Marks IM, Shear MK, Greist JM. The work and social adjustment scale: a simple measure of impairment in functioning. Br J Psychiatry (2002) 180(05):461–4. doi: 10.1192/bjp.180.5.461

PubMed Abstract | CrossRef Full Text | Google Scholar

51. McGuire-Snieckus R, McCabe R, Catty J, Hansson L, Priebe S. A new scale to assess the therapeutic relationship in community mental health care: STAR. Psychol Med (2007) 37(1):85–95. doi: 10.1017/S0033291706009299

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Health Research Authority. Research Ethics Service. Research Ethics Service and Research Ethics Committees. [cited 2019 Nov 5]. Available from: https://www.hra.nhs.uk/about-us/committees-and-services/res-and-recs/.

Google Scholar

53. R core Team. R: A language and environment for statistical computing (2017). https://www.R-project.org/

Google Scholar

54. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw (2015) 67(1):1–48. doi: 10.18637/jss.v067.i01

CrossRef Full Text | Google Scholar

55. Kuznetsova A, Brockhoff PB, Christensen RHB. lmerTest Package: tests in linear mixed effects models. J Stat Softw (2017) 82(13):1–26. doi: 10.18637/jss.v082.i13

CrossRef Full Text | Google Scholar

56. Lenth RV. Least-squares means: the R package lsmeans. J Stat Softw (2016) 69(1):1–33. doi: 10.18637/jss.v069.i01

CrossRef Full Text | Google Scholar

57. Elliott R, Timulak L. Descriptive and interpretive approaches to qualitative research. In: Miles J, Gilbert P, editors. A handbook of research methods for clinical and health psychology. England: Oxford University Press (2005). p. 147–60. Available from: https://pdfs.semanticscholar.org/159b/3cab28e0d5d7fa9a54469737cf391e65cabe.pdf.

Google Scholar

58. Jarrett RB, Minhajuddin A, Gershenfeld H, Friedman ES, Thase ME. Preventing depressive relapse and recurrence in higher-risk cognitive therapy responders. JAMA Psychiatry (2013) 70(11):1152. doi: 10.1001/jamapsychiatry.2013.1969

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Sheets ES, Wilcoxon Craighead L, Brosse AL, Hauser M, Madsen JW, Edward Craighead W. Prevention of recurrence of major depression among emerging adults by a group cognitive-behavioral/interpersonal intervention. J Affect Disord (2013) 147(1–3):425–30. doi: 10.1016/j.jad.2012.08.036

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Wang PS, Aguilar-Gaxiola S, Alonso J, Angermeyer MC, Borges G, Bromet EJ, et al. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet (2007) 370(9590):841–50. doi: 10.1016/S0140-6736(07)61414-7

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res (2014) 16(6):e146. doi: 10.2196/jmir.3077

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Richards D, Timulak L, Doherty G, Sharry J, Colla A, Joyce C, et al. Internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression: protocol for a randomized controlled trial. BMC Psychiatry (2014) 14(1):147. doi: 10.1186/1471-244X-14-147

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of Mental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Ment Heal (2018) 5(1):e20. doi: 10.2196/mental.9769

CrossRef Full Text | Google Scholar

64. Community and Mental Health Team. Annual report on the use of IAPT services, England 2017-18. NHS, England (2018). Available from: https://files.digital.nhs.uk/52/D3168F/psych-ther-2017-18-ann-rep.pdf.

Google Scholar

65. Marshall D, Quinn C, Child S, Shenton D, Pooler J, Forber S, et al. What IAPT services can learn from those who do not attend. J Ment Heal (2016) 25(5):410–5. doi: 10.3109/09638237.2015.1101057

CrossRef Full Text | Google Scholar

66. Clark DM, Canvin L, Green J, Layard R, Pilling S, Janecka M. Transparency about the outcomes of mental health services (IAPT approach): an analysis of public data. Lancet (2018) 391(10121):679–86. doi: 10.1016/S0140-6736(17)32133-5

PubMed Abstract | CrossRef Full Text | Google Scholar

67. Erbe D, Eichert H-C, Riper H, Ebert DD. Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: systematic review. J Med Internet Res (2017) 19(9):e306. doi: 10.2196/jmir.6588

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Kok RN, Van Straten A, Aartjan M, Beekman TF, Cuijpers P. Short-Term Effectiveness of Web-Based Guided Self-Help for Phobic Outpatients: Randomized Controlled Trial. (2014) [cited 2019 Nov 8]; Available from: http://www.webcitation.org/6O2ioOQSs.

Google Scholar

69. Kenter RMF, van de Ven PM, Cuijpers P, Koole G, Niamat S, Gerrits RS, et al. Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interv (2015) 2(1):77–83. doi: 10.1016/j.invent.2015.01.001

CrossRef Full Text | Google Scholar

70. Braamse AMJ, van Meijel B, Visser OJ, Boenink AD, Cuijpers P, Eeltink CE, et al. A randomized clinical trial on the effectiveness of an intervention to treat psychological distress and improve quality of life after autologous stem cell transplantation. Ann Hematol (2016) 95(1):105–14. doi: 10.1007/s00277-015-2509-6

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Earley C, Joyce C, McElvaney J, Richards D, Timulak L. Preventing depression: qualitatively examining the benefits of depression-focused iCBT for participants who do not meet clinical thresholds. Internet Interv (2017) 9:82. doi: 10.1016/j.invent.2017.07.003

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Richards D, Murphy T, Viganó N, Timulak L, Doherty G, Sharry J, et al. Acceptability, satisfaction and perceived efficacy of “Space from Depression” an internet-delivered treatment for depression. Internet Interv (2016) 5:12–22. doi: 10.1016/j.invent.2016.06.007

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Jasper K, Weise C, Conrad I, Andersson G, Hiller W, Kleinstäuber M. The working alliance in a randomized controlled trial comparing Internet-based self-help and face-to-face cognitive behavior therapy for chronic tinnitus. Internet Interv (2014) 1(2):49–57. doi: 10.1016/j.invent.2014.04.002

CrossRef Full Text | Google Scholar

74. Newton J, Sundin EC. A questionnaire-based qualitative study of therapist views on computerized CBT. Cognit Behav Ther (2016) 9:e15. doi: 10.1017/S1754470X16000131

CrossRef Full Text | Google Scholar

75. Beatty L, Binnion C, Kemp E, Koczwara B. A qualitative exploration of barriers and facilitators to adherence to an online self-help intervention for cancer-related distress. Support Care Cancer (2017) 25(8):2539–48. doi: 10.1007/s00520-017-3663-2

PubMed Abstract | CrossRef Full Text | Google Scholar

76. MacLeod M, Martinez R, Williams C. Cognitive behaviour therapy self-help: who does it help and what are its drawbacks? Behav Cognit Psychother (2009) 37(01):61. doi: 10.1017/S1352465808005031

CrossRef Full Text | Google Scholar

77. Sinclair C, Holloway K, Riley G, Auret K. Online mental health resources in rural Australia: clinician perceptions of acceptability. J Med Internet Res (2013) 15(9):e193. doi: 10.2196/jmir.2772

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Bell CJ, Colhoun HC, Carter FA, Frampton CM. Effectiveness of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust NZ J Psychiatry (2012) 46(7):630–40. doi: 10.1177/0004867412437345

CrossRef Full Text | Google Scholar

79. Learmonth D, Trosh J, Rai S, Sewell J, Cavanagh K. The role of computer-aided psychotherapy within an NHS CBT specialist service. Couns Psychother Res (2008) 8(2):117–23. doi: 10.1080/14733140801976290

CrossRef Full Text | Google Scholar

80. Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of emental health for mood disorders in routine practice: Systematic review of barriers and facilitating factors. J Med Internet Res (2018) 20(3). doi: 10.2196/preprints.9769

CrossRef Full Text | Google Scholar

81. Andersson G, Titov N, Dear BF, Rozental A, Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry (2019). doi: 10.1002/wps.20610

CrossRef Full Text | Google Scholar