The therapeutic alliance in cognitive-behavioral therapy for obsessive-compulsive disorder: A systematic review and meta-analysis

Background: The therapeutic alliance has been recognized as one of the most researched key elements of treatment across different therapeutic approaches and diagnostic domains. Despite its importance, our current understanding of its clinical relevance in patients with obsessive-compulsive disorder (OCD) is still debated. This study aimed to examine empirical evidence on the effect of alliance on treatment outcomes in Cognitive Behavioral Therapy (CBT) in patients with OCD in a systematic review and meta-analysis.

Methods: Original peer-reviewed articles until March 2022 were included if they were (1) written in English; (2) included a clinical group with a current primary OCD diagnosis; (3) involved individual CBT; (4) used a validated therapeutic alliance scale that was related to the outcome measurement; (5) reported an effect size.

Results: Thirteen studies were included, six of which contained sufficient statistical information to be included in the meta-analysis. A total of 897 patients took part in all reviewed studies. We found a modest effect of alliance on post-treatment outcome [Tau² = −0.1562 (C.I. 95%: −0.2542 to −0.0582)].

Discussion: The results show the existence of considerable variability and methodological inconsistencies across studies. We discuss the role of methodological factors that could account for this divergence, the research limitations, and the implications for current research.

Systematic review registration: [https://osf.io/dxez5/?view_only=bc2deaa7f0794c8dbef440255b2d4b3b].

Introduction

Obsessive-Compulsive Disorder (OCD) is a serious mental health condition characterized by recurrent and persistent thoughts, urges, or images that are experienced as highly disturbing and intrusive (obsessions) and/or stereotyped recurrent mental or physical behaviors aimed to ignore or neutralize them (compulsions) (1). OCD symptom domains typically include contamination obsession and washing/cleaning compulsion; obsession concerning responsibility for harm, injury, or bad luck and checking compulsion; unacceptable obsessional thoughts concerning sex, violence, or religion associated with mental neutralizing strategies; obsession about symmetry, completeness, and exactness and ordering compulsion (2). Given the complexity and heterogeneity of symptoms, several genetic, behavioral, and cognitive models have been proposed to explain the mechanisms behind this spectrum [e.g., (3–6)].

The estimated lifetime prevalence of the full disorder is approximately 2–3%, with most individuals with OCD being affected before their mid-twenties (7, 8). OCD shows a chronic course, and it is highly comorbid with anxiety disorders and major depressive disorder (9). In the absence of effective treatment, OCD results in significant distress, functional impairments in social and occupational functioning, and reduced quality of life. Therefore, it is considered a disabling mental health condition associated with significant personal and socio-economic costs (10).

A combined approach that includes cognitive therapy and behavioral therapy represents the currently recommended psychological treatment of choice for OCD (11), showing the highest degree of empirical support in meta-analytic investigations [e.g., (12–14)]. According to NICE guidelines, it is recommended a “stepped care” model, with increasing intensity of treatment according to clinical severity and complexity (2005). This treatment includes exposure with response prevention (EX/RP) with or without OCD-focused cognitive therapy (CT). EX/RP is a behavioral therapy that comprises the implementation of a series of in-session and between-session exposures that are planned and implemented through collaboration between patient and therapist. The treatment generally includes more or less prolonged exposure to obsessional triggers and procedures aimed at blocking rituals. Although the optimal frequency of sessions has not been defined, both intensive, which involves daily sessions over 1 month, and weekly sessions, have been proved effective in reducing symptoms (15, 16). Compared to medications alone, EX/RP protocol is more effective with a lower relapse rate (17). However, despite the effectiveness of such structured, evidence-based treatment, up to 18.7% of OCD patients will drop out prior to completion of treatment (18). Further, about 50% of OCD patients still complain about some residual symptoms even after successful treatment, with a negative impact on their quality of life (19–21). Indeed, a significant proportion of OCD patients receiving cognitive and behavioral therapies is subjected to post-treatment relapse, with an estimated full recovery rate of approximately 25% (22). Despite the undoubtedly high impact that CT and EX/RP exert on patients, there is still a significant degree of variability associated with treatment response. This variance cannot be fully explained by the effect of the specific treatment and probably needs to be accounted for by other variables. Therefore, identifying the complex factors associated with successful treatment outcomes is crucial to optimizing the delivery of evidence-based psychotherapeutic interventions.

In the last decades, research in the field of psychotherapy has increasingly focused on examining potential mechanisms of therapeutic change, aiming to identify predictors of treatment response. In this context, the therapeutic relationship variables received particular attention, especially in the operationalized construct of the alliance. From a transtheoretical perspective, the alliance can be defined as a collaborative stance between client and therapist, underpinned by three components: (a) consensus on therapeutic goals, (b) agreement on therapeutic tasks, and (c) a positive bond between client and therapist (23). The therapeutic alliance is historically defined as a “non-specific” interpersonal factor auxiliary to technical procedures that produce change (24). Cognitive-behavioral perspective emphasizes the collaborative nature of the therapeutic alliance. Within this framework, the alliance is conceptualized as a necessary but not sufficient therapeutic change factor (25, 26), allowing the creation of trust and safety conditions between patient and therapist that, in turn, facilitate the application of specific techniques.

Nevertheless, the therapeutic alliance is widely recognized as a crucial component of treatment across all psychotherapeutic approaches. Accordingly, a substantial number of empirical research have been addressed to explore the association between alliance and post-treatment outcomes. In this respect, the recent Third Interdivisional APA Task Force on Evidence-Based Relationships and Responsiveness synthesized empirical studies investigating the association between the therapeutic relationship and outcome and suggested alliance as a “demonstrably effective” ingredient of the therapeutic change process across treatments and diagnoses (27). On the whole, literature accumulated so far suggest that the alliance is moderately associated with treatment outcomes in a transdiagnostic way (28–33), yet its impact on process change has been sparsely investigated in specific disorders. Indeed, most experimental and meta-analytic studies estimate the alliance-outcome association by aggregating disorders and treatments. However, evidence suggesting potential differences across disorders also exists, with some diagnostic groups being more affected by therapeutic alliance than others (34, 35). For instance, the alliance seems to have less impact on severe anxiety disorders, substance abuse, and eating disorders than on other disorders, such as depression (27, 32, 36–38). Notably, investigating how the alliance works for specific disorders has been delineated as one of the key questions for future studies by the Third Interdivisional APA Task Force on Evidence-Based Relationships and Responsiveness (27).

In this respect, no systematic review to date has been specifically aimed at exploring the alliance-outcome relationship in individuals receiving Cognitive-Behavioral Therapy (CBT) for OCD.

Although OCD is no longer categorized as an anxiety disorder in the DSM based on significant diagnostic validators, it is important to highlight that a substantial overlap between OCD and the anxiety disorders is still acknowledged by many clinicians and researchers [e.g., (39)]. For this reason, some recent reviews pulled together OCD with disorders like post-traumatic stress disorder, generalized anxiety disorder, and social anxiety disorder. Specifically, two recent qualitative reviews on the role of therapeutic alliance in anxiety-related disorders (32, 33) added a contribution to this research area. In their critical review, Buchholz and Abramowitz (32) provided an overview of existing research on the alliance-outcome relationship in exposure therapy for anxiety-related disorders, including OCD. Results suggest a link between a strong alliance and symptom reduction in EX/RP therapy for OCD, with some evidence indicating that task and goal alliance dimensions, relative to the bond alliance, were the strongest predictors of post-treatment outcome, along with treatment adherence. Importantly, this critical literature review also revealed substantial methodological and conceptual differences among investigations, including alliance assessment tools, timing and perspective of the alliance assessment, and diagnoses (32). Accordingly, in a subsequent critical review (33), the alliance-outcome relationship in CBT for anxiety disorders was also found to differ significantly across the timing of the alliance assessment (e.g., early, middle, and late assessment in the course of therapy), perspectives of the alliance rater (e.g., patient, therapist or observer-rated alliance), and specific alliance dimensions. However, it is important to note that the alliance-outcome relationship in OCD was not the primary focus of investigation in these recent literature reviews; further, both reviews restricted their critical analysis to adult samples and face-to-face therapies, potentially limiting the generalizability of the results to younger individuals with OCD as well as electronically delivered treatments such as internet-based CBT (iCBT).

Collectively, recent findings suggest a role of the alliance, particularly task and goal alliance dimensions, in predicting post-treatment outcomes in anxiety-related disorders, including OCD. Crucially, the literature accumulated so far has produced mixed findings potentially due to significant methodological and conceptual differences among studies (32, 33). Therefore, the effective role of the therapeutic alliance as a change mechanism in CBT for individuals with OCD remains unclear. The present systematic review and meta-analysis aimed to synthesize the available empirical studies investigating the relationship between therapeutic alliance and post-treatment outcomes with CBT in patients with OCD. Further clarifying the impact of the therapeutic alliance in the psychotherapeutic approach for individuals with OCD could enrich our understanding of effective therapeutic change factors implementing evidence-based treatments for this diagnostic group.

Materials and methods

The systematic review process was conducted according to the PRISMA guidelines (40–43)¹ and preregistered on Open Science Framework (OSF).² The PRISMA protocol consists of a 27-item checklist and a 4-phase flow diagram that guides the systematic review process (see Figure 1).

FIGURE 1

Figure 1. Flowchart of selection process for included articles.

Research strategies

We conducted a systematic search of articles published in peer-reviewed journals articled indexed in the following electronic databases: PubMed (1949 to March 2022), Scopus (1788 to March 2022), PsycINFO (1806 to March 2022), PsychArticles (1800 to March 2022), and Web of Science (1900 to March 2022). The search strategy used Boolean combinations of the following keywords: (“obsessive-compulsive disorder” OR “OCD”) AND (“therapeutic relationship” OR “working relationship” OR “collaborative relationship” OR “alliance” OR “working alliance” OR “helping alliance” OR “therapist factors” OR “mediator” OR “emotional bond” OR “alliance-outcome relationship”). Mendeley reference manager software³ was used to import the references from the databases and to remove duplicates. The first screening was made by reading the title and abstract by the authors F.S. and V.S. The same authors read the full text of the selected studies. In addition to systematic searches in the above databases, we also searched for additional articles in the reference lists of the selected papers (i.e., backward research) and identified studies that cited the selected articles (i.e., forward research).

Eligibility criteria

According with our aims (i.e., investigating the relation between therapeutic alliance and outcome in OCD), we included studies that fulfilled the following criteria: (a) original, peer-reviewed articles; (b) written in English; (c) included a clinical group with a current primary OCD diagnosis; (d) were empirical and included quantitative data (i.e., reviews, case studies, and qualitative papers were excluded); (e) involved individual CBT; (f) used a validated therapeutic alliance scale with adequate psychometric properties (specifically, we included only studies that used the scales recommended by Martin et al. (31) and Elvins and Green (44) as core measures of the construct); (g) measured the relationship between therapeutic alliance and at least one systematic outcome measure (Yale-Brown Obsessive Compulsive Scale, Y-BOCS; Obsessive Compulsive Inventory, OCI) in the context of individual CBT for OCD; (h) a reported effect (d or r), its equivalent (standardized β weight), or other statistic (t or F) that could be converted to an effect. Articles from all publication years were accepted.

Data collection

Descriptive and quantitative data extraction was performed from each study and included: (a) metadata (i.e., authors and year of publication); (b) information related to the sample (i.e., sample size, age, gender, and onset age); (c) methodological information (i.e., alliance and OCD scales, alliance and outcome rater(s), the timing of the assessment); (d) main results and effects size (see Table 1).

TABLE 1

Table 1. Synthetic description of studies that have examined the influence of therapeutic alliance on treatment outcome.

Moreover, all the articles were screened according to the research criteria for process research proposed by Lemmens et al. (45) and employed by Baier et al. (46) in a systematic review to assess the quality of the studies. These criteria comprised: (a) the use of a randomized controlled trial (RCT) design, (b) the use of a control group, (c) sample size defined as n ≥ 40, (d) the inclusion of different mediators in the design and statistical analysis, (e) assessment at two or more time points of alliance (not averaged during the analyses), and (f) experimental manipulation of the construct of the alliance.

The author F.S. performed a quality check and accuracy of the author’s V.S. data extraction. Inter-rater kappa reliability between the two coders was excellent (κ = 0.95), and minimal differences in coding were resolved through discussion till the agreement became perfect.

Meta-analysis procedure

For the articles included in the systematic review, additional exclusion criteria were considered for conducting the meta-analysis in order to improve comparability between studies. Specifically, we included the studies that reported: (a) standardized β weights smaller than 0.5 or bigger than −0.5; (b) direct effect analysis of the relationship between the therapeutic relationship and outcome (see paragraph Statistical Analyses for more details).

Statistical analyses

For each study, we extracted effect sizes computed as correlation coefficients or standardized β weights between therapeutic alliance and treatment outcome measures. Methodological considerations suggest that β weights should not be used as surrogates for correlation coefficients because they reflect the influence of the predictor variables in a multiple regression model (47). Thus, we approximated the bivariate Pearson correlation using the standardized regression coefficients as suggested by Peterson and Brown (48). The standardized β weights (that fell within an interval between −0.5 and 0.5) were transformed in Pearson correlation using the formula:

$\text{r}=\mathrm{\beta }+0.05\mathrm{\lambda }$

where λ equals 1 when β is non-negative and 0 when β is negative.

The authors have shown that the relationship between r and β appears robust and independent of sample size and the number of predictor variables when within this interval. Indeed, they reported that “it is possible to derive a formula for imputing an r value missing assuming a knowledge of a corresponding β weight” because there is “a relatively tight joint distribution of β and r values within the range from −0.50 to 0.50” (48). To compute this formula we used the algorithm suited by the Practical Meta-Analysis Effect Size Calculator⁴ (49)].

In studies with more than one outcome measure, we averaged correlations or standard weights using the arithmetic mean to obtain one effect for each study, to avoid over-representing multi-analyses studies in the following analysis.

Then, each correlation was converted in Cohen’s d using the conversion software Psychometrica [Calculation of Effect Sizes,⁵ Dettelbach (Germany): Psychometrica]. By convention, an effect size of 0.2 is considered small, a value of 0.5 is moderate, and a value of 0.8 or greater is considered a relatively large effect (50).

The meta-analyses were performed with Jamovi 2 (MAJOR module following procedures suggested by Borenstein et al. (51) and Cooper (52). All analyses were carried out using the Fisher r-to-z transformed correlation coefficient as the outcome measure. We started by fitting a random-effects model to the data (53) and estimated the amount of heterogeneity (i.e., tau²) with the restricted maximum-likelihood estimator (54). In addition to the estimate of tau², the Q-test for heterogeneity (55) and the I² statistic were computed. In case any amount of heterogeneity was detected (i.e., tau² > 0, regardless of the results of the Q-test), the software provided a prediction interval for the true outcomes.

To assess whether studies may be outliers and/or influential in the context of the model, we used studentized residuals and Cook’s distances (56). Studies with a studentized residual larger than the 100 × [1–0.05/(2 × k)]^th percentile of a standard normal distribution were considered potential outliers (i.e., using a Bonferroni correction with two-sided alpha = 0.05 for k studies included in the meta-analysis). Studies with a Cook’s distance larger than the median plus six times the interquartile range of the Cook’s distances are considered influential.

We computed the rank correlation test and the regression symmetry test to assess publication bias using the standard error of the observed outcomes as the predictor, and we created a funnel plot (57).

Results

Study selection

The literature search strategy and inclusion criteria yielded 847 studies that measured the relationship between the therapeutic alliance and post-treatment outcome in subjects with OCD. As shown in Figure 1, the literature search generated 802 potentially relevant articles (after 35 duplicates removal).

After titles and abstract screening, 763 were excluded. The full text of the remaining 39 eligible studies was retrieved and reviewed; articles were excluded either because they did not meet the inclusion criteria, or were qualitative studies, reviews, or commentaries.

This screening resulted in the inclusion of 13 articles for review. Studies were published between 1989 and 2022 and conducted in the United States or Europe. The sample sizes ranged from 17 to 208 for the patients and from 9 to 42 for the therapists. Most of the studies did not report the number of therapists who took part in the study. The average age ranged from 13 to 40 years old, and 55% were women. All articles reported that the participants had a diagnosis of OCD. In total, 897 patients took part in all studies reviewed [(58,59) employed the same sample of patients; thus, the sample was counted only once]. Eight studies reported the average age of onset, which varied from 16.81 to 34.8 years old.

Regarding the criteria for process research (45, 46), we note that seven studies (54%) were part of an RCT design in which different treatments were tested, and six had a control group. Eight studies (61%) had a sample size greater than 40 patients, and three (25%) employed multiple mediators in the experimental design. Finally, four studies (30%) assessed alliance more than in two-time points, and none did an experimental manipulation of the alliance between patient and therapist (Table 1).

Of the 13 selected articles, eight reported a relationship between therapeutic alliance and the outcome measure. Three studies found that patient adherence fully mediated the relationship, one found mixed results, and another did not find any relationship (Table 2).

TABLE 2

Table 2. Studies meeting criteria for process research.

Obsessive-compulsive disorder subtypes

Although the cardinal features of OCD are obsessions and compulsions, a variety of clinically significant obsessive-compulsive symptoms (e.g., checking, excessive washing, and ordering) may meet the diagnostic criteria for OCD. This pattern of heterogeneity that includes the age of onset (early vs. late onset), patterns of comorbidity, and presenting symptoms have been associated with different subtypes schemes presumably underlying different etiologies and neural correlates [e.g., (60–62)].

Only two articles provided information about OCD subtypes in the selected sample. Specifically, Maher et al. (59) reported that 4% of the sample belonged to the hoarding subtype; Keijsers et al. (63) reported that 53% of patients belonged to the checking subtype, 15% to the washing, 17% to the checking and washing, and 15% had obsessions only. However, no study investigated the relationship between the OCD subtype and alliance.

Intervention/treatment

Most of the studies employed EX/RP intervention in various formats, from short and intensive [daily and lasting < 4 weeks; (64)] to more standard plans with weekly or twice-weekly sessions lasting between 4 and 8 weeks (58, 59, 65–71). Among these studies, four used a combined CBT protocol: the Pediatric OCD Treatment (64, 72), a CBT program that included both group and dyadic psychotherapy sessions (69), and the web-based ICBT (65, 68). All combined programs included a mix of psychoeducation, cognitive training, and EX/RP. Finally, one study employed also Stress Management Training [SMT; (73)] based on exposure with cessation of compulsions (58), one used the EX/RP augmented by motivational interviewing (MI) strategies (70), while another adopted the Inference Based Approach (IBA) (71).

Overall, all studies administered the EX/RP protocol either as a stand-alone treatment or in combination with other CBT interventions, and all patients received individual treatment.

Outcome measurement

Symptom severity and treatment response were evaluated by the clinician-rated Yale-Brown Obsessive Compulsive Scale [Y-BOCS; (74, 75)], a semi-structured interview regarded as the “gold standard” in the measurements of OCD symptoms (obsessions and compulsions in the last week) (58, 59, 65–71, 76). Most of the studies employed independent evaluators for the rating, i.e., clinical psychologists, blind to treatment conditions and treatment outcomes. Only one study adopted both the clinician and self-report versions [Y-BOCS-SR; (69)]. Two studies also administered the Obsessive-Compulsive Inventory revised [OCI-R; 68, 70, 77)], and one used the Maudsley Obsessive-Compulsive Inventory [MOCI; (78)]. Both OCI and MOCI are self-report measures that assess the distress associated with obsessions and compulsions. Finally, only the earliest reviewed study adopted patients’ self-monitoring as an outcome measurement (79).

Despite some consistency in the scales used for outcome evaluation, studies considerably varied in the measurement timing. Two studies included only one time-point assessment at the end of the treatment (67, 68), two studies measured before and after the treatment (66, 71), six studies used three or four timepoints at baseline, mid-treatment, and at the end of the treatment (58, 59, 63, 64, 70, 76), and one study measured the symptoms with the self-report Y-BOCS at the end of each week (69). Finally, one study also took measurements at different time points during follow-up at 3, 6, 9, and 12 months (76). As it will be later discussed, the number of time points in which the outcome is measured represents an essential parameter for studying the reciprocal influence between symptom change and alliance.

Although symptom change represents an important parameter and a general index of treatment success, it might not be sufficient to depict the psychological wellness in the patient’s daily life. Indeed, an individual’s functioning presumably depends on both symptom severity and symptom management. Thus, assessing the quality of life in relation to therapeutic alliance and post-treatment outcomes seems to provide a complementary measurement. However, only one of the reviewed studies assessed the quality of life using the Quality of Life Enjoyment and Satisfaction Questionnaire [QLESQ; (80)], a self-report measure administered at different time points, and it did not find any relation between alliance and QLESQ (70).

Finally, as suggested by Buchholz and Abramowitz (32), another important variable to relate to the alliance is patients’ dropout during treatment. Recent studies seem to suggest a positive relationship between therapeutic alliance and patient retention [cf. review of (81–84)]. Despite its importance, only a few studies reported the number of patients that dropped the treatment, which was, in any case, a low rate.

Alliance measurement

Five different measures of the therapeutic alliance, in varying formats, were used in these studies.

Most of the reviewed studies assessed the alliance with the Working Alliance Inventory [WAI; (85)], a self-report inventory that was originally designed to measure Bordin’s working alliance dimensions (bond, task, and goals). Five studies administered the patient-rated version (58, 59, 65–67), three used both the patient and therapist versions (64, 70, 71), and one did not specify which version was used (68). Among these studies, some employed the standard version composed of 36 items (58, 59, 64), while others adopted the short forms composed of 12 items by Tracey and Kokotovic (65, 67, 68, 70, 86) or by Hatcher and Gillaspy (66, 68, 87). Keeley et al. (64), that assessed the therapeutic alliance in a pediatric population, also administered the WAI to the caregivers and the Therapeutic Alliance Scale for Children-Revised to the patients [TASC-R; (88)].

The remaining studies assessed the quality of alliance using the self-report Barret-Lennard Relationship Inventory [RI, (89)] (79), the items related to the bond of the Helping Alliance Questionnaire HAq (90, 76), and the alliance subscale of the Bern Post-Session Report (91). These scales differ from the WAI in several aspects, such as the dimensions of the alliance that are represented and the number of items that the measures contain. For instance, the RI, a measure of empathy, correlates only with the WAI bond scale but not with the task and goals dimensions (44). Meta-analytic results have shown that the different versions of the same scale (e.g., long or short format and different rater versions) differed in predicting treatment outcomes (30). Despite some similarities and shared themes between the alliance scales, they do not have a common account of the alliance construct (92, 93). In particular, no scale has a complete representation of the different properties belonging to the concept of alliance that has been proposed in the past years.

Another potential confounding factor in the alliance-outcome relationship is the variability across alliance raters. Ideally, a good measure of the alliance should have a good consistency, measured as inter-rater reliability (94). Horvath et al. (30) estimated that the variables “type of measure” and “raters” account for 23% of the variance in predicting the treatment outcome. However, the variability across raters per se does not seem to represent a strong methodological issue, given the moderate correlation between patients’ and therapists’ alliance scores (31). Nevertheless, the reviewed studies showed mixed results in predicting treatment outcomes among alliance raters. For instance, Keeley et al. (64) and Hoogduin et al. (79) found that only therapist rating predicted treatment outcome when the alliance was measured in an early stage. However, both therapist and patient predicted the treatment outcome when the alliance was measured in a mid-phase. Conversely, Strauss et al. (70) found that only patient averaged alliance scores covaried with outcome treatment, but patient and therapist early scores were not associated with symptom change. Some mixed results were also shown in the study published by Keijsers et al. (63), where patient ratings correlated with a reduction in obsessive fear; however, only therapist ratings classified patients as success or treatment failure with multivariate analysis. Finally, Wolf et al. (71) showed that only therapist rating measured in an early phase significantly predicted the post-treatment outcome. Taken together, these results show that therapist and patient ratings differed in how they predicted treatment outcomes across studies. Thus, it would be helpful to relate these differences across raters in predicting symptom change with some inter-rater reliability measure (95). Indeed, only one study reported a measure of consistency across raters and found a weak significant correlation (71), although most recent studies reported overall good reliability of alliance scores measured with Cronbach’s alpha.

Albeit, the therapeutic alliance has been described as an intrinsic dyadic concept that involves a process of mutual influence and impact between the therapist and the patient [e.g., (23, 96–98)], 50% of the studies reviewed here asked only patients to rate alliance. This limitation reflects the main trend in therapeutic alliance studies that mostly focus on patients’ views only (30). However, recent studies have highlighted the importance of taking into account both patient and therapist perspectives, introducing the concept of patient-therapist alliance “congruence” (i.e., the inter-rater agreement on alliance quality at one time-point) and the alliance “convergence” (i.e., the degree of change over time in the inter-rater agreement on alliance quality) (99–101). These constructs reflect the dynamic nature of this dyadic process and carry complementary information on a therapeutic relationship that is asymmetric by nature (102).

Wide variability across studies was also observed in the evaluation timing. Indeed, seven studies collected data only at one time-point, either in the early (58, 59, 63, 66, 67, 71, 76), mid (76), or late phase of the treatment (68). Two studies used three or four time-points at baseline, mid-treatment, and at the end of the treatment (64, 70), and one study measured alliance through all treatment at the end of each week (69).

Alliance-outcome relationship measurement

All studies reviewed here assessed the relationship between alliance and some measure of symptom outcome. The two earliest studies computed the correlation between alliance and self-report symptom outcome (79) or obsessive fear symptom change (63). Both studies found evidence of a positive effect of alliance on the treatment outcome. However, these results should be taken with caution because of the less rigorous methodology employed.

Subsequent studies investigated the role of the alliance as a mediator of symptom change with more sophisticated statistical analyses, such as a causal stepwise approach using linear regressions (103) and structural equation modeling (104). Other methods included growth analysis (105) and the longitudinal mixed-effects model (106). Overall, the nine studies that used a regression approach found evidence of alliance as a predictor of positive change. However, three of eight found that the general effect of alliance [(58, 59), with an overlapping sample] or task alliance (66) on treatment outcome was fully mediated by patient’s adherence to between- and within-session exposure tasks. Among the most recent studies having the highest frequency of ratings of both alliance and symptoms (i.e., at the end of each week or throughout all treatment), no alliance’s influence on symptom decrease was found (69, 70). The only study that compared simultaneous and cross-lag models to assess the effect of reciprocal influence in the alliance-outcome relationship found that patient alliance covaried with symptoms change as measured with the Y-BOCS; however, changes in previous Y-BOCS scores predicted subsequent changes in alliance scores, thus suggesting that an improvement in the reported symptoms precedes the changes in the alliance scores (70).

Furthermore, among the studies that assessed alliance on patients alone, one followed the data analysis suggested by Baldwin et al. (107). This analysis consists of decomposing alliance-outcome correlations into two components: the “within-therapist correlations” at the patient level (i.e., how the alliance is related to outcome in each therapist) and “between-therapist correlations” at the therapist level (i.e., how the alliance is related to outcome across therapists). Importantly, this method allows computing the cross-level interaction between patients’ and therapists’ variability (107). In the study that implemented this analytic method, it was found that therapist variability in the task/goal dimension of the alliance predicted treatment outcome, while patient variability in the alliance did not. Conversely, the therapeutic bond was not related to the outcome.

Overall, this qualitative analysis of the reviewed studies points to some indication of a positive relationship between alliance and treatment outcome. Nevertheless, given the high degree of methodological variance even in the most recent studies that employed more advanced statistical and experimental design, these results should be interpreted in the context of this variability across studies.

Meta-analysis

The purpose of this section of the review is to quantify the present literature regarding the existence and strength of the relation between therapeutic alliance and outcome in the context of OCD. As the preceding qualitative analysis illustrated, there is considerable variability across studies with regard to how and when alliance and outcome were measured, who were the raters, and how the relationship alliance-outcome was measured. Thus, to improve comparability among the results, some exclusion criteria were applied that narrowed down to k = 6 the number of studies that were included in the analyses (64–67, 71, 76). In particular, one study was excluded because it included self-monitoring as outcome measure and did not use a validated alliance scale (79); the others did not report any statistics that could be used in the meta-analysis (59, 63, 68–70, 73). All the pooled effect sizes reflected patient’s rating.

We first performed a model with the global alliance score (task, goal, and bond). In the selected studies (64–67, 71), the observed Fisher r-to-z transformed correlation coefficients ranged from −0.3541 to −0.0902, with most estimates being negative (100%). The estimated average Fisher r-to-z transformed correlation coefficient based on the random-effects model was Tau² = −0.1562 (C.I.95%: −0.2542 to −0.0582; Figure 2). Therefore, the average outcome differed significantly from zero (z = −3.1231, p = 0.0018). According to the Q-test, there was no significant amount of heterogeneity in the true outcomes [Q(4) = 2.3834, p = 0.6656, tau² = 0.0000, I² = 0.0000%]. Hence, even though there may be some heterogeneity, the true outcomes of the studies were generally in the same direction as the estimated average outcome.

FIGURE 2

Figure 2. Forest plot visualizing the relationship between the global therapeutic alliance and the treatment outcome for each included study. Horizontal bars show 99% confidence intervals, with the study having a significant effect denoted by horizontal bars that do not touch the dotted vertical line (the line of no effect). Diamonds sizes reflect the weight of the overall study.

An examination of the studentized residuals revealed that none of the studies had a value larger than ± 2.5758; hence there was no indication of outliers in the context of this model. According to Cook’s distances, none of the studies could be considered overly influential. Neither the rank correlation nor the regression test indicated any funnel plot asymmetry (p = 0.8167 and p = 0.3780, respectively; see Figure 3).

FIGURE 3

Figure 3. Funnel plot. Estimates (z-score) from selected studies (on the horizontal axes) plotted against each study’s standard error (on the vertical axes).

Next, we pooled the effect sizes for the task/goal alliance dimension and for the bond dimension separately. For the task/goal dimension, in the included studies (66, 67, 71), the transformed correlation coefficient based on the random-effects model was Tau² = −0.1977 (95% CI: −0.3149 to −0.0805). Therefore, the average outcome differed significantly from zero (z = −3.3071, p = 0.0009; Supplementary Figure 1). According to the Q-test, there was no significant amount of heterogeneity in the true outcomes [Q(2) = 1.9141, p = 0.3840, tau² = 0.0000, I² = 0.0244%].

For the bond dimension, in the included studies (66, 67, 71, 76), the transformed correlation coefficient based on the random-effects model was Tau² = −0.1372 (95% CI: −0.3669 to −0.0924). Therefore, the average outcome did not differ significantly from zero (z = −1.1710, p = 0.2416; Supplementary Figure 2). According to the Q-test, the true outcomes appear to be heterogeneous [Q(3) = 9.2209, p = 0.0417, tau² = 0.0351, I² = 66.2000%]. A 95% prediction interval for the true outcomes is given by −0.5703 to 0.2958. Hence, although the average outcome is estimated to be negative, in some studies the true outcome may in fact be positive. This amount of heterogeneity might depend on pooling effect sizes from two different scales (WAI and HAq). Although both scales measure the bond dimension and are presumably correlated, they cannot be traced back to a common second-order factor.

Discussion

This systematic review aimed to synthesize the current scientific evidence on the relationship between therapeutic alliance and treatment outcome in CBT in patients with OCD.

Overall, we found a modest association between alliance and treatment outcome. This result is consistent with previous systematic reviews and meta-analyses that quantified this relationship with an aggregate r of 0.28 across disorders and treatments (27) or found a positive association between alliance and symptom change in anxiety-related disorders (32, 33). In particular, our results suggest that task and goal alliance domains, which may be more characterized as cognitive factors (32), are associated with post-treatment outcomes, while the therapeutic bond is not. Most of the included studies measured alliance in the early or mid-phase of the treatment. Thus, it is possible that agreeing on the goals and the willingness to be engaged in the exposure tasks might be predictive factors in the first sessions of the therapy.

We also found substantial variability of different sources across studies. Hoogduin et al. (79) and Keeley et al. (64) reported that both early- and mid-treatment therapist alliance, but only mid-treatment patient alliance, was associated with an improved outcome in adult and pediatric populations, respectively. Consistently, Vogel et al. (76) and Andersson et al. (65) reported that mid patient alliance was associated with better outcomes. Conversely, Keijsers et al. (63) found that early patient alliance but not therapist alliance was associated with improvement in obsessive fears (though not to compulsive behaviors). Similarly, Hagen et al. (67) found that early patient task and goal alliance (though not bond) was associated with a better outcome, while Wheaton found that only early patient task alliance predicted post-treatment outcome. Also, Wolf et al. (71) found that early task and goal alliance and early therapist alliance (total score) predicted the post-treatment outcome. Differently from these studies that reported a positive association between alliance and outcome, one study found that overall early patient alliance was not related to outcome (69). Moreover, one of the most recent studies found a mixed pattern of results, and only in EX/RP (not in SMT) symptom change was associated with subsequent changes in alliance (though not vice versa) (70). Finally, three studies found that overall early patient alliance was mediated by adherence (58, 59, 66).

All the reviewed studies differ in many methodological aspects, such as outcome and alliance measurements, alliance raters and timing, and statistical approaches. Most recent studies which adopted more sophisticated analyses (e.g., structural equation, linear growth analyses, stepwise regression analysis, and cross-lagged model), took into account early symptom improvement and examined temporal associations of the alliance over time, are also showing less or no evidence of alliance affecting treatment outcome. Thus, our results should be interpreted in the context of this variability, and several limitations should be considered. For instance, half of the studies employed RCTs to examine differences across treatment, and used a control group. Second, 58% of studies used a sample size equal to or greater than 40, that is generally considered the minimum number to have sufficient power in the meta-analysis (108, 109). However, our meta-analysis, which had an average sample size of 66 participants, is sufficiently powered to detect small effects [1-β err prob 0.99, formula retrieved by (110)]. Third, three studies disentangled the effect of alliance from other factors, such as symptom severity, adherence to treatment, quality of life, patient’s expectancy, and motivation, and OCD sub-types (e.g., washing, hoarding). Therefore, more mediation studies are needed to assess how alliance contributes to mediate the relationship between treatment and symptom change and to separate its effect from other specific and generic factors (e.g., age, gender, comorbidity with other disorders, age of onset, use of psychiatric medication, patient’s and therapist’s factors). Fourth, the small number of available studies and the fact that some of them were carried out by some overlapping research groups (and presumably with similar programs and methodology), although mostly with different samples, makes it difficult to reach firm conclusions. Fifth, only four studies assessed alliance at two or more time-points, not averaging during the analyses (64, 69, 70, 79).

Temporality represents an essential parameter in experimental designs investigating the therapeutic alliance. While in the past, process-based research tended to represent the association between alliance and outcome as a static “snapshot” by taking measurements at only one time-point or by averaging, the inherently dynamic nature of the therapeutic interaction is now being increasingly recognized [e.g., (111)]. This change of view, which implies studying how alliance changes over the course of treatment through cycles of ruptures and repairs (112), and how it interacts with the specific treatment, entails achieving a fuller picture of this dynamic dyadic process. Indeed, a key factor in studying this phenomenon implies understanding the direction of the alliance-outcome link. A positive link might have at least three different sources: (a) alliance produces patient’s symptoms change; (b) symptom change induces a change in the therapeutic alliance between therapist and patient; (c) alliance and treatment outcome influence each other (113). To test the idea that alliance influences treatment outcome, the alliance must predict outcome as measured at a subsequent time point, taking into account potential changes in the outcome preceding alliance measurement (114). Thus, a gold standard for future studies would be to collect repeated measurements of alliance and outcome (after and during session) and combine them with more sophisticated statistical approaches (e.g., cross-lagged panel model) to analyze the interactions and reciprocal influences between these variables over time. Among the reviewed studies Schwartz et al. (69), had the highest sampling rate, assessing alliance and outcome at the end of each session/week. On the other hand Strauss et al. (70), compared simultaneous and cross-lagged models to study the reciprocal influences between alliance and outcome but sampling fewer time-points.

Another important factor to consider is the different impact of therapeutic alliance domains on symptoms change. Indeed, three studies that disentangled the effect of goal, task and bond, found that treatment outcome was significantly predicted by goal and task alliance but not by bond (66, 67, 71). This result could be associated with the specific EX/RP techniques that are used for the treatment of OCD, that require setting appropriate goals, and providing tasks that allow patients to go far enough in exposures to the fearful stimuli or situations (115). It is possible to argue that such a structured and challenging treatment protocol, for both therapist and patient, creates the basis for building an alliance that relies more on tasks and goals consensus than on emotional attachment and more in general on the feelings and attitudes that patient and therapist have toward each other. It is also possible to speculate that bond might represent a more significant outcome predictor in therapies with less structured protocols.

Finally, it must be noted that a significant association between alliance and outcome does not imply a significant clinical impact. For instance, Wolf et al. (71) found that although therapeutic alliance significantly predicted post-treatment outcome, it accounted for only 2% of symptoms improvement, resulting in alliance being not clinically relevant in their sample. This result might be due to the high rates in the alliance scores that makes it difficult to reach higher scores and thus a stronger association between alliance and outcome (71). Future studies need to take into account alliance scores variability and possible ceiling effects that might hinder the effect of alliance on post-treatment outcome.

Some limitations should be considered when interpreting these results. First, in this review, there is a possible language bias since the research strategy was limited to articles published in English, and we also did not include unpublished studies. Therefore, it is possible that some relevant papers were missed. Also, since studies with samples of different ages were included, it was not possible to draw conclusions on specific populations. Moreover, most of the studies in the current review were based on research protocols that included structured interventions and highly trained therapists; thus, the generalizability of these results to more naturalistic settings (e.g., general community practice and patients with comorbidities) remains unclear. Naturalistic studies are required to demonstrate whether alliance works in the field with patients with OCD. Finally, for the statistical analyses, we used the approach proposed by Peterson and Brown (48) to convert the standardized β weights into Pearson correlation coefficients. Although this approach has been widely used in the literature because it provides a straightforward method to deal with missing values, it has some limitations. Recently, it has been shown that this approach can lead to an underestimation of meta-analytic correlations and that the estimated correlations do not perform better than using existing correlations (116, 117). Thus, our meta-analytic results should be interpreted in the context of this trade-off between generalization and approximation.

In summary, the present review shows some evidence of an interactive effect between alliance and the treatment outcome in individuals with OCD, although with considerable variability in reporting varying measurement time-points, experimental designs, statistical approaches, measurements tools, and alliance raters. However, we sought to achieve the greatest possible consistency in our data extraction and meta-analysis.

Future studies that include more refined temporal assessment of alliance, larger samples, and measures of potentially interacting variables are required to better understand whether changes in alliance interact with treatment response or vice versa and which is the clinical impact of this association. Understanding this complex relationship will ultimately help to improve outcomes for individuals living with OCD.

Data availability statement

The original contributions presented in this study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author/s.

Author contributions

AMS, GG, and FM conceived the study. FS and VS reviewed the literature, analyzed the data, and wrote the first version of the draft. GA supervised the statistical analyses. GD’A, TD, and NP contributed to the discussion of the literature. All authors contributed to the writing and editing of the draft.

Funding

This study was supported by the Italian Ministry of Education, University and Research.

Acknowledgments

We sincerely thank Nadja Wolf for sharing statistical details about published data and all the co-authors, Patricia van Oppen, Adriaan W. Hoogendoorn, Anton J. L. M. van Balkom, and Henny A. D. Visser.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.951925/full#supplementary-material

Footnotes

1. ^ https://www.bmj.com/content/372/bmj.n71
2. ^ https://osf.io/dxez5/
3. ^ https://www.mendeley.com/reference-management/reference-manager
4. ^ https://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-SMD22.php
5. ^ https://www.psychometrica.de/effect_size.html

References

1. American Psychiatric Association [APA] Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Washington, DC: American Psychiatric Association (2013). doi: 10.1176/appi.books.9780890425596

CrossRef Full Text | Google Scholar

2. Abramowitz JS, Deacon BJ, Olatunji BO, Wheaton MG, Berman NC, Losardo D, et al. Assessment of obsessive-compulsive symptom dimensions: development and evaluation of the dimensional obsessive-compulsive scale. Psycnet APA Org. (2010) 22:180–98. doi: 10.1037/a0018260

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Salkovskis PM. Obsessional-compulsive problems: a cognitive-behavioural analysis. Behav Res Ther. (1985) 23:571–83. doi: 10.1016/0005-7967(85)90105-6

CrossRef Full Text | Google Scholar

4. Rachman S. A cognitive theory of obsessions. In: E Sanavio editor. Behavior And Cognitive Therapy Today. Oxford: Pergamon (1998). p. 209–22.

Google Scholar

5. Foa EB, Franklin ME. Cognitiveâ behavioral treatment of obsessiveâ compulsive disorder. In: DK Routh, RJ DeRubeis editors. The Science Of Clinical Psychology: Accomplishments And Future Directions. Washington, DC: American Psychological Association (1998). p. 235–63.

Google Scholar

6. Mancini F. The Obsessive Mind: Understanding and Treating Obsessive-Compulsive Disorder. New York, NY: Routledge (2018). doi: 10.4324/9780429452956

CrossRef Full Text | Google Scholar

7. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. (2005) 62:617–27. doi: 10.1001/archpsyc.62.6.617

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Brakoulias V, Starcevic V, Belloch A, Brown C, Ferrao YA, Fontenelle LF, et al. Comorbidity, age of onset and suicidality in obsessive–compulsive disorder (OCD): an international collaboration. Compre Psychiatry. (2017) 76:79–86. doi: 10.1016/J.COMPPSYCH.2017.04.002

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Ruscio AM, Stein DJ, Chiu WT, Kessler RC. The epidemiology of obsessive-compulsive disorder in the National comorbidity survey replication. Mol Psychiatry. (2010) 15:53–63. doi: 10.1038/mp.2008.94

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Eaton WW, Martins SS, Nestadt G, Bienvenu OJ, Clarke D, Alexandre P. The burden of mental disorders. Epidemiol Rev. (2008) 2008:1–14. doi: 10.1093/epirev/mxn011

PubMed Abstract | CrossRef Full Text | Google Scholar

11. National Collaborating Centre for Mental Health [NCCMH]. Obsessive-Compulsive Disorder: Core Interventions in the Treatment of Obsessive-Compulsive Disorder and Body Dysmorphic Disorder. London: NICE (2005).

Google Scholar

12. Rosa-Alcázar AI, Sánchez-Meca J, Gómez-Conesa A, Marín-Martínez F. Psychological treatment of obsessive–compulsive disorder: a meta-analysis. Clin Psychol Rev. (2008) 28:1310–25. doi: 10.1016/J.CPR.2008.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Olatunji BO, Davis ML, Powers MB, Smits JAJ. Cognitive-behavioral therapy for obsessive–compulsive disorder: a meta-analysis of treatment outcome and moderators. J Psychiatr Res. (2013) 47:33–41. doi: 10.1016/J.JPSYCHIRES.2012.08.020

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Öst LG, Havnen A, Hansen B, Kvale G. Cognitive behavioral treatments of obsessive–compulsive disorder. A systematic review and meta-analysis of studies published 1993–2014. Clin Psychol Rev. (2015) 40:156–69. doi: 10.1016/J.CPR.2015.06.003

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Storch EA, Merlo LJ, Bengtson M, Murphy TK, Lewis MH, Yang MC, et al. D-cycloserine does not enhance exposure-response prevention therapy in obsessive-compulsive disorder. Int Clin Psychopharmacol. (2007) 22:230–7. doi: 10.1097/YIC.0B013E32819F8480

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Franklin ME, Foa EB. Cognitive behavioral treatments for obsessive compulsive disorder. In: PE Nathan, JM Gorman editors. A Guide To Treatments That Work. Oxford: Oxford University Press (2002). p. 367–86.

Google Scholar

17. Abramowitz J, Franklin M, Foa E. Empirical status of cognitive-behavioral therapy for obsessive-compulsive disorder: a meta-analytic review. Rom J Cogn Behav Psychother. (2010) 2002:89–104. doi: 10.1016/j.cpr.2005.07.003

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Ong CW, Clyde JW, Bluett EJ, Levin ME, Twohig MP. Dropout rates in exposure with response prevention for obsessive-compulsive disorder: What do the data really say? J Anxiety Disorders. (2016) 40:8–17. doi: 10.1016/j.janxdis.2016.03.006

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Loerinc A, Meuret A, Twohig MP, Rosenfield D, Bluett EJ, Craske MG. Response rates for CBT for anxiety disorders: need for standardized criteria. Clin Psychol. (2015) 42:72–82. doi: 10.1016/j.cpr.2015.08.004

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Fontenelle IS, Fontenelle LF, Borges MC, Prazeres AM, Rangé BP, Mendlowicz MV, et al. Quality of life and symptom dimensions of patients with obsessive–compulsive disorder. Psychiatry Res. (2010) 179:198–203. doi: 10.1016/J.PSYCHRES.2009.04.005

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Key BL, Rowa K, Bieling P, McCabe R, Pawluk EJ. Mindfulness-based cognitive therapy as an augmentation treatment for obsessive–compulsive disorder. Clin Psychol Psychother. (2017) 24:1109–20. doi: 10.1002/CPP.2076

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Fisher PL, Wells A. How effective are cognitive and behavioral treatments for obsessive–compulsive disorder? A clinical significance analysis. Behav Res Ther. (2005) 43:1543–58. doi: 10.1016/J.BRAT.2004.11.007

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy. (1979) 16:252–60. doi: 10.1037/H0085885

CrossRef Full Text | Google Scholar

24. Castonguay LG. “Common factors” and “nonspecific variables”: clarification of the two concepts and recommendations for research. J Psychother Integr. (1993) 3:267–86. doi: 10.1037/H0101171

CrossRef Full Text | Google Scholar

25. Beck AT. Cognitive Therapy of Depression. New York, NY: Guilford press (1979).

Google Scholar

26. DeRubeis RJ, Brotman MA, Gibbons CJ. A conceptual and methodological analysis of the nonspecifics argument. Clin Psychol. (2005) 12:174. doi: 10.1093/clipsy.bpi022

CrossRef Full Text | Google Scholar

27. Norcross JC, Lambert MJ. Psychotherapy relationships that work III. Psychotherapy. (2018) 55:303–15. doi: 10.1037/PST0000193

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Browne J, Mueser KT, Meyer-Kalos P, Gottlieb JD, Estroff SE, Penn DL. The therapeutic alliance in individual resiliency training for first episode psychosis: relationship with treatment outcomes and therapy participation. J Consult Clin Psychol. (2019) 87:734–44. doi: 10.1037/CCP0000418

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Flückiger C, Del AC, Wampold BE, Horvath AO. The alliance in adult psychotherapy: a meta-analytic synthesis. Psychotherapy. (2018) 55:316–40. doi: 10.1037/PST0000172

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Horvath AO, Del Re AC, Flückiger C, Symonds D. Alliance in individual psychotherapy. In: JC Norcross editor. Psychotherapy Relationships That Work: Evidence-Based Responsiveness. Oxford: Oxford University Press (2011). p. 25–69. doi: 10.1093/ACPROF:OSO/9780199737208.003.0002

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Martin DJ, Garske JP, Katherine Davis M. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol. (2000) 68:438–50. doi: 10.1037/0022-006X.68.3.438

CrossRef Full Text | Google Scholar

32. Buchholz JL, Abramowitz JS. The therapeutic alliance in exposure therapy for anxiety-related disorders: a critical review. J Anxiety Disord. (2020) 70:102194. doi: 10.1016/J.JANXDIS.2020.102194

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Luong HK, Drummond SPA, Norton PJ. Elements of the therapeutic relationship in CBT for anxiety disorders: a systematic review. J Anxiety Disord. (2020) 76:102322. doi: 10.1016/J.JANXDIS.2020.102322

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Beckner V, Vella L, Howard I, Mohr DC. Alliance in two telephone-administered treatments: relationship with depression and health outcomes. J Consult Clin Psychol. (2007) 75:508–12. doi: 10.1037/0022-006X.75.3.508

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Bedics JD, Atkins DC, Harned MS, Linehan MM. The therapeutic alliance as a predictor of outcome in dialectical behavior therapy versus nonbehavioral psychotherapy by experts for borderline personality disorder. Psychotherapy. (2015) 52:67–77. doi: 10.1037/A0038457

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Krupnick JL, Sotsky SM, Elkin I, Simmens S, Moyer J, Watkins J, et al. The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: findings in the national institute of mental health treatment of depression collaborative research program. FOCUS. (2006) 4:269–77. doi: 10.1176/FOC.4.2.269

CrossRef Full Text | Google Scholar

37. Flückiger C, Del Re AC, Wampold BE, Symonds D, Horvath AO. How central is the alliance in psychotherapy? A multilevel longitudinal meta-analysis. J Couns Psychol. (2012) 59:10. doi: 10.1037/a0025749

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Graves TA, Tabri N, Thompson-Brenner H, Franko DL, Eddy KT, Bourion-Bedes S, et al. A meta-analysis of the relation between therapeutic alliance and treatment outcome in eating disorders. Int J Eat Disord. (2017) 50:323–40. doi: 10.1002/EAT.22672

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Nemeroff CB, Weinberger D, Rutter M, MacMillan HL, Bryant RA, Wessely S, et al. DSM-5: a collection of psychiatrist views on the changes, controversies, and future directions. BMC Med. (2013) 11:202. doi: 10.1186/1741-7015-11-202/PEER-REVIEW

CrossRef Full Text | Google Scholar

40. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. (2009) 62:e1–34. doi: 10.1016/j.jclinepi.2009.06.006

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group*. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Internal Med. (2009) 151:264–9. doi: 10.7326/0003-4819-151-4-200908180-00135

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. (2015) 4:1–9. doi: 10.1186/2046-4053-4-1

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. (2021) 88:105906.

Google Scholar

44. Elvins R, Green J. The conceptualization and measurement of therapeutic alliance: an empirical review. Clin Psychol Rev. (2008) 28:1167–87. doi: 10.1016/j.cpr.2008.04.002

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Lemmens LH, Galindo-Garre F, Arntz A, Peeters F, Hollon SD, DeRubeis RJ, et al. Exploring mechanisms of change in cognitive therapy and interpersonal psychotherapy for adult depression. Behav Res Ther. (2017) 94:81–92. doi: 10.1016/j.brat.2017.05.005

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Baier AL, Kline AC, Feeny NC. Therapeutic alliance as a mediator of change: a systematic review and evaluation of research. Clin Psychol Rev. (2020) 82:101921. doi: 10.1016/j.cpr.2020.101921

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Hunter JE, Schmidt FL. Dichotomization of continuous variables: the implications for meta-analysis. J Appl Psychol. (1990) 75:334. doi: 10.1037/0021-9010.75.3.334

CrossRef Full Text | Google Scholar

48. Peterson RA, Brown SP. On the use of beta coefficients in meta-analysis. J Appl Psychol. (2005) 90:175. doi: 10.1037/0021-9010.90.1.175

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Wilson SJ, Lipsey MW. Wilderness challenge programs for delinquent youth: a meta-analysis of outcome evaluations. Eval Prog Plan. (2000) 23:1–12. doi: 10.1016/S0149-7189(99)00040-3

CrossRef Full Text | Google Scholar

50. Sawilowsky SS. New effect size rules of thumb. J Modern Appl Stat Methods. (2009) 8:26.

Google Scholar

51. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to Meta-Analysis. Hoboken, NJ: John Wiley & Sons (2021). doi: 10.1002/9781119558378

CrossRef Full Text | Google Scholar

52. Cooper H. Research Synthesis and Meta-Analysis: A Step-by-Step Approach. (Vol. 2). Thousand oaks, CA: Sage publications (2015).

Google Scholar

53. Raudenbush SW. Analyzing effect sizes: random-effects models. Handbook Res Synth Meta Anal. (2009) 2:295–316.

Google Scholar

54. Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. (2005) 30:261–93. doi: 10.3102/10769986030003261

CrossRef Full Text | Google Scholar

55. Cochran WG. The combination of estimates from different experiments. Biometrics. (1954) 10:101–29. doi: 10.2307/3001666

CrossRef Full Text | Google Scholar

56. Tabachnick BG, Fidell LS, Ullman JB. Using Multivariate Statistics. (Vol. 5). Boston, MA: Pearson (2007). p. 481–98.

Google Scholar

57. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. (1997) 315(7109):629–34. doi: 10.1136/bmj.315.7109.629

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Simpson HB, Maher MJ, Wang Y, Bao Y, Foa EB, Franklin M. Patient adherence predicts outcome from cognitive behavioral therapy in obsessive-compulsive disorder. J Consult Clin Psychol. (2011) 79:247–52. doi: 10.1037/a0022659

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Maher MJ, Wang Y, Zuckoff A, Wall MM, Franklin M, Foa EB, et al. Predictors of patient adherence to cognitive-behavioral therapy for obsessive-compulsive disorder. Psychother Psychosomat. (2012) 81:124. doi: 10.1159/000330214

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Taylor S. Early versus late onset obsessive–compulsive disorder: evidence for distinct subtypes. Clin Psychol Rev. (2011) 31:1083–100. doi: 10.1016/j.cpr.2011.06.007

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Endres D, Pollak TA, Bechter K, Denzel D, Pitsch K, Nickel K, et al. Immunological causes of obsessive-compulsive disorder: is it time for the concept of an “autoimmune OCD” subtype? Transl Psychiatry. (2022) 12:1–14. doi: 10.1038/s41398-021-01700-4

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Han S, Xu Y, Guo HR, Fang K, Wei Y, Liu L, et al. Two distinct subtypes of obsessive compulsive disorder revealed by heterogeneity through discriminative analysis. Hum Brain Mapp. (2022) 43:3037–46. doi: 10.1002/hbm.25833

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Keijsers GP, Hoogduin CA, Schaap CP. Predictors of treatment outcome in the behavioural treatment of obsessive-compulsive disorder. Br J Psychiatry. (1994) 165:781–6. doi: 10.1192/bjp.165.6.781

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Keeley ML, Geffken GR, Ricketts E, McNamara JP, Storch EA. The therapeutic alliance in the cognitive behavioral treatment of pediatric obsessive–compulsive disorder. J Anxiety Disord. (2011) 25:855–63. doi: 10.1016/j.janxdis.2011.03.017

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Andersson E, Ljótsson B, Hedman E, Enander J, Kaldo V, Andersson G, et al. Predictors and moderators of Internet-based cognitive behavior therapy for obsessive-compulsive disorder: Results from a randomized trial. J Obsessive Compul Relat Disord. (2015) 4:1–7. doi: 10.1016/j.jocrd.2014.10.003

CrossRef Full Text | Google Scholar

66. Wheaton MG, Huppert JD, Foa EB, Simpson HB. How important is the therapeutic alliance in treating obsessive-compulsive disorder with exposure and response prevention? An empirical report. Clin Neuropsychiatry. (2016) 13:88–93.

Google Scholar

67. Hagen K, Solem S, Opstad HB, Vogel PA, Kennair LEO, Kvale G, et al. Therapist variability in the task/goal dimension of the early working alliance predicts outcome in exposure and response prevention treatment for obsessive-compulsive disorder. Clin Neuropsychiatry. (2016) 13:94–9.

Google Scholar

68. Herbst N, Franzen G, Voderholzer U, Thiel N, Knaevelsrud C, Hertenstein E, et al. Working alliance in internet-based cognitive-behavioral therapy for obsessive-compulsive disorder. Psychother Psychosomat. (2016) 85:117–8. doi: 10.1159/000441282

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Schwartz C, Hilbert S, Schubert C, Schlegl S, Freyer T, Löwe B, et al. Change factors in the process of cognitive-behavioural therapy for obsessive–compulsive disorder. Clin Psychol Psychother. (2017) 24:785–92. doi: 10.1002/cpp.2045

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Strauss AY, Huppert JD, Simpson HB, Foa EB. What matters more? Common or specific factors in cognitive behavioral therapy for OCD: therapeutic alliance and expectations as predictors of treatment outcome. Behav Res Ther. (2018) 105:43–51. doi: 10.1016/j.brat.2018.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Wolf N, van Oppen P, Hoogendoorn AW, van Balkom AJLM, van Visser HAD. Therapeutic alliance and treatment outcome in cognitive behavior therapy for obsessive-compulsive disorder. Front Psychiatry. (2022) 24:658693. doi: 10.3389/fpsyt.2022.658693

PubMed Abstract | CrossRef Full Text | Google Scholar

72. POTS Study Team. Cognitive-behavior therapy, sertraline, and their combination for children and adolescents with obsessive-compulsive disorder: the Pediatric OCD Treatment Study (POTS) randomized controlled trial. JAMA. (2004) 292:1969–76. doi: 10.1001/jama.292.16.1969

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Simpson HB, Foa EB, Liebowitz MR, Ledley DR, Huppert JD, Cahill S, et al. A randomized, controlled trial of cognitive-behavioral therapy for augmenting pharmacotherapy in obsessive-compulsive disorder. Am J Psychiatry. (2008) 165:621–30. doi: 10.1176/appi.ajp.2007.07091440

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Goodman WK, Price LH, Rasmussen SA, Mazure C, Delgado P, Heninger GR, et al. The yale-brown obsessive compulsive scale: II. Validity. Arch Gen Psychiatry. (1989) 46:1012–6. doi: 10.1001/archpsyc.1989.01810110054008

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Storch EA, Rasmussen SA, Price LH, Larson MJ, Murphy TK, Goodman WK. Development and psychometric evaluation of the yale–brown obsessive-compulsive scale—second edition. Psychol Assess. (2010) 22:223. doi: 10.1037/a0018492

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Vogel PA, Hansen B, Stiles TC, Götestam KG. Treatment motivation, treatment expectancy, and helping alliance as predictors of outcome in cognitive behavioral treatment of OCD. J Behav Ther Exp Psychiatry. (2006) 37:247–55. doi: 10.1016/j.jbtep.2005.12.001

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Foa EB, Huppert JD, Leiberg S, Langner R, Kichic R, Hajcak G, et al. The obsessive-compulsive inventory: development and validation of a short version. Psychol Assess. (2002) 14:485. doi: 10.1037/1040-3590.14.4.485

CrossRef Full Text | Google Scholar

78. Hodgson RJ, Rachman S. Obsessional-compulsiv complaints. Behav Res Ther. (1977) 15:389–95. doi: 10.1016/0005-7967(77)90042-0

CrossRef Full Text | Google Scholar

79. Hoogduin CAL, De Haan E, Schaap CPDR. The significance of the patient—therapist relationship in the treatment of obsessive-compulsive neurosis. Br J Clin Psychol. (1989) 28:185–6. doi: 10.1111/j.2044-8260.1989.tb00831.x

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Endicott J, Nee J, Harrison W, Blumenthal R. Quality of life enjoyment and satisfaction questionnaire: a new measure. Psychopharmacol Bull. (1993) 29:321–6. doi: 10.1037/t49981-000

CrossRef Full Text | Google Scholar

81. Sharf J, Primavera LH, Diener MJ. Dropout and therapeutic alliance: a meta-analysis of adult individual psychotherapy. Psychotherapy. (2010) 47:637–645. doi: 10.1037/a0021175

PubMed Abstract | CrossRef Full Text | Google Scholar

82. Roos J, Werbart A. Therapist and relationship factors influencing dropout from individual psychotherapy: a literature review. Psychother Res. (2013) 23:394–418. doi: 10.1080/10503307.2013.775528

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Huber J, Jennissen S, Nikendei C, Schauenburg H, Dinger U. Agency and alliance as change factors in psychotherapy. J Consult Clin Psychol. (2021) 89:214–26. doi: 10.1037/ccp0000628

PubMed Abstract | CrossRef Full Text | Google Scholar

84. Sijercic I, Liebman RE, Stirman SW, Monson CM. The effect of therapeutic alliance on dropout in cognitive processing therapy for posttraumatic stress disorder. J Trauma Stress. (2021) 34:819–28. doi: 10.1002/jts.22676

PubMed Abstract | CrossRef Full Text | Google Scholar

85. Horvath AO, Greenberg LS. Development and validation of the Working Alliance Inventory. J Counsel Psychol. (1989) 36:223. doi: 10.1037/0022-0167.36.2.223

CrossRef Full Text | Google Scholar

86. Tracey TJ, Kokotovic AM. Factor structure of the working alliance inventory. Psychol Assess. (1989) 1:207. doi: 10.1037/1040-3590.1.3.207

CrossRef Full Text | Google Scholar

87. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the Working Alliance Inventory. Psychother Res. (2006) 16:12–25. doi: 10.1590/2237-6089-2019-0099

PubMed Abstract | CrossRef Full Text | Google Scholar

88. Creed TA, Kendall PC. Therapist alliance-building behavior within a cognitive-behavioral treatment for anxiety in youth. J Consult Clin Psychol. (2005) 73:498. doi: 10.1037/0022-006X.73.3.498

PubMed Abstract | CrossRef Full Text | Google Scholar

89. Barrett-Lennard GT. Dimensions of therapist response as causal factors in therapeutic change. Psychol Monogr. (1962) 76:1–36. doi: 10.1186/s13054-016-1208-6

PubMed Abstract | CrossRef Full Text | Google Scholar

90. Luborsky L. Therapeutic alliances as predictors of psychotherapy outcomes: factors. Working Alliance. (1994) 173:38.

Google Scholar

91. Flückiger C, Regli D, Zwahlen D, Hostettler S, Caspar F. Der berner patienten-und therapeutenstundenbogen 2000. Z Klinische Psychol Psychother. (2010) 39:71–9. doi: 10.1026/1616-3443/a000015

CrossRef Full Text | Google Scholar

92. Hatcher RL, Barends AW. Patients’ view of the alliance in psychotherapy: exploratory factor analysis of three alliance measures. J Consult Clin Psychol. (1996) 64:1326. doi: 10.1037/0022-006X.64.6.1326

PubMed Abstract | CrossRef Full Text | Google Scholar

93. Hatcher RL. Alliance theory and measurement. In: JC Muran, JP Barber editors. The Therapeutic Alliance: An Evidence-Based Guide to Practice. New York, NY: The Guilford Press (2010). p. 7–28.

Google Scholar

94. Hunsley J, Mash EJ. Developing criteria for evidence-based assessment: an introduction to assessments that work. (2008):3–14. doi: 10.1093/med:psych/9780195310641.003.0001

CrossRef Full Text | Google Scholar

95. Hunsley J, Mash EJ. Evidence-based assessment. Annu. Rev. Clin. Psychol. (2007) 3:29–51. doi: 10.1146/annurev.clinpsy.3.022806.091419

PubMed Abstract | CrossRef Full Text | Google Scholar

96. Greenson RR. The working alliance and the transference neurosis. Psychoanal Q. (2008) 77:77–102. doi: 10.1002/j.2167-4086.2008.tb00334.x

PubMed Abstract | CrossRef Full Text | Google Scholar

97. Li X, O’Connor S, Kivlighan DM Jr., Hill CE. “Where is the relationship” revisited: using actor-partner interdependence modeling and common fate model in examining dyadic working alliance and session quality. J Counsel Psychol. (2021) 68:194. doi: 10.1037/cou0000515

PubMed Abstract | CrossRef Full Text | Google Scholar

98. Soma CS, Baucom BR, Xiao B, Butner JE, Hilpert P, Narayanan S, et al. Coregulation of therapist and client emotion during psychotherapy. Psychother Res. (2020) 30:591–603.

Google Scholar

99. Laws HB, Constantino MJ, Sayer AG, Klein DN, Kocsis JH, Manber R, et al. Convergence in patient–therapist therapeutic alliance ratings and its relation to outcome in chronic depression treatment. Psychother Res. (2017) 27:410–24. doi: 10.1080/10503307.2015.1114687

PubMed Abstract | CrossRef Full Text | Google Scholar

100. Coyne AE, Constantino MJ, Laws HB, Westra HA, Antony MM. Patient–therapist convergence in alliance ratings as a predictor of outcome in psychotherapy for generalized anxiety disorder. Psychother Res. (2018) 28:969–84. doi: 10.1080/10503307.2017.1303209

PubMed Abstract | CrossRef Full Text | Google Scholar

101. Nissen-Lie HA, Solbakken OA, Falkenström F, Wampold BE, Holmqvist R, Ekeblad A, et al. Does it make a difference to be more “on the same page”? Investigating the role of alliance convergence for outcomes in two different samples. Psychother Res. (2021) 31:573–88. doi: 10.1080/10503307.2020.1823030

PubMed Abstract | CrossRef Full Text | Google Scholar

102. Markin RD. Toward a common identity for relationally oriented clinicians: a place to hang one’s hat. Psychotherapy. (2014) 51:327. doi: 10.1037/a0037093

PubMed Abstract | CrossRef Full Text | Google Scholar

103. Baron RM, Kenny DA. The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. (1986) 51:1173. doi: 10.1037/0022-3514.51.6.1173

PubMed Abstract | CrossRef Full Text | Google Scholar

104. Preacher KJ, Hayes AF. SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behav Res Methods Instruments Comput. (2004) 36:717–31. doi: 10.3758/BF03206553

PubMed Abstract | CrossRef Full Text | Google Scholar

105. Cheong J, MacKinnon DP, Khoo ST. Investigation of mediational processes using parallel process latent growth curve modeling. Struct Equat Model. (2003) 10:238–62. doi: 10.1207/S15328007SEM1002_5

PubMed Abstract | CrossRef Full Text | Google Scholar

106. Curran PJ, Bollen KA. The best of both worlds: combining autoregressive and latent curve models. In: LM Collins, AG Sayer editors. ew Methods for the Analysis of Change. Washington, DC: American Psychological Association (2001). p. 107–35. doi: 10.1037/10409-004

CrossRef Full Text | Google Scholar

107. Baldwin SA, Wampold BE, Imel ZE. Untangling the alliance-outcome correlation: exploring the relative importance of therapist and patient variability in the alliance. J Consult Clin Psychol. (2007) 75:842. doi: 10.1037/0022-006X.75.6.842

PubMed Abstract | CrossRef Full Text | Google Scholar

108. Hedges LV, Pigott TD. The power of statistical tests for moderators in meta-analysis. Psychol Methods. (2004) 9:426. doi: 10.1037/1082-989X.9.4.426

PubMed Abstract | CrossRef Full Text | Google Scholar

109. Kazdin AE, Bass D. Power to detect differences between alternative treatments in comparative psychotherapy outcome research. J Consult Clin Psychol. (1989) 57:138. doi: 10.1037/0022-006X.57.1.138

PubMed Abstract | CrossRef Full Text | Google Scholar

110. Valentine JC, Pigott TD, Rothstein HR. How many studies do you need? A primer on statistical power for meta-analysis. J Educ Behav Stat. (2010) 35:215–47. doi: 10.3102/1076998609346961

CrossRef Full Text | Google Scholar

111. Aron L, Harris A. Relational Psychoanalysis: Evolution of Process. (Vol. 5). London: Routledge (2014). doi: 10.4324/9781315786995

CrossRef Full Text | Google Scholar

112. Safran JD, Muran JC. Has the concept of the therapeutic alliance outlived its usefulness? Psychotherapy. (2006) 43:286. doi: 10.1037/0033-3204.43.3.286

PubMed Abstract | CrossRef Full Text | Google Scholar

113. Barber JP, Connolly MB, Crits-Christoph P, Gladis L, Siqueland L. Alliance predicts patients’ outcome beyond in-treatment change in symptoms. J Consult Clin Psychol. (2000) 68:1027. doi: 10.1037/0022-006X.68.6.1027

PubMed Abstract | CrossRef Full Text | Google Scholar

114. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry. (2002) 59:877–83. doi: 10.1001/archpsyc.59.10.877

PubMed Abstract | CrossRef Full Text | Google Scholar

115. Gillihan SJ, Williams MT, Malcoun E, Yadin E, Foa EB. Common pitfalls in exposure and response prevention (EX/RP) for OCD. J Obsessive Compul Relat Disord. (2012) 1:251–7. doi: 10.1016/j.jocrd.2012.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

116. Aloe AM, Thompson CG. The synthesis of partial effect sizes. J Soc Soc Work Res. (2013) 4:390–405. doi: 10.5243/jsswr.2013.24

CrossRef Full Text | Google Scholar

117. Roth PL, Le H, Oh IS, Van Iddekinge CH, Bobko P. Using beta coefficients to impute missing correlations in meta-analysis research: reasons for caution. J Appl Psychol. (2018) 103:644–58. doi: 10.1037/apl0000293

PubMed Abstract | CrossRef Full Text | Google Scholar