Psychometric properties of comprehensive cognitive, affective, and psychomotor competency assessment scales in psychodynamic psychotherapy for borderline personality disorder

Background: Psychodynamic psychotherapy is a type of psychotherapy for individuals with borderline personality disorder (BPD). However, competency in conducting effective psychodynamic psychotherapy for BPD is difficult to evaluate. Therefore, this study aimed to identify the psychometric properties of a comprehensive scale to assess cognitive, affective, and psychomotor competencies (CS-CAPC) in psychodynamic psychotherapy for BPD.

Methods: This is a qualitative study. The first step used the Delphi technique to gather experts’ opinions on the cognitive, affective, and psychomotor competencies necessary to conduct psychodynamic psychotherapy for BPD. The experts comprised three psychotherapists, seven psychiatrists with experience in psychotherapy, and nine teaching staff. A panel discussion was conducted to obtain qualitative data. Thematic data analysis was adopted, and content validity testing was used to analyze the content validity of the CS-CAPC in psychodynamic psychotherapy for BPD.

Results: The CS-CAPC comprised two scales assessing two specific competencies in psychodynamic psychotherapy for BPD: The first scale, the psychodynamic formulation competency assessment scale (PF-CAS), comprised six items, including the case description, etiology, and potential course of therapy. The second scale, the practical-competency assessment scale (PC-CAS) for psychodynamic psychotherapy for BPD, comprised 12 items, including building a therapeutic alliance, performing psychodynamic interventions while working through the therapeutic process, and closing the session. The scale-level content validity index (S-CVI) for the PF-CAS was 0.981, and that for the PC-CAS in psychodynamic psychotherapy for BPD was 1.00.

Conclusion: The CS-CAPC in psychodynamic psychotherapy for BPD had good validity in assessing individual competency in the cognitive, affective, and psychomotor domains.

1 Introduction

Borderline personality disorder (BPD) is difficult to treat. Patients with BPD have complex symptoms that can make them uncooperative, extremely sensitive, and exhibit alternating positive and negative emotions that often inflict intense negative emotional reactions on therapists, such as frustration, anger, or feeling incompetent (1). Prevalence studies suggest that the prevalence of BPD in the general population is approximately 1.6% to 1.8%. The prevalence of this disorder is higher in psychiatric outpatient populations (approximately 11%) and reaches 20% in psychiatric inpatient population (2–4). In Indonesia, although no nationwide data are available on the prevalence of BPD, there appears to be an increase in the number of patients with BPD, according to data from the psychiatric clinic and ward of Indonesia’s national referral center, Dr. Cipto Mangunkusumo National Central General Hospital (RSUPN-CM), Jakarta, Indonesia. In 2020, 242 clinic visits were recorded from 71 patients, including four new patients. Additionally, 25 cases of hospitalization of patients with BPD were recorded. In 2021, the number of polyclinic visits and hospitalizations increased to 775 clinic visits and 190 patients, including 26 new patients in polyclinics and 68 inpatient hospitalization cases (5).

Psychotherapy and psychopharmacology are both used to assist individuals with BPD (6). Psychodynamic psychotherapy can be applied as a psychotherapeutic approach, whose main goals in individuals with BPD include improving self-cohesion, self-other representations, affect regulation, and mentalization capacity by helping them attain a better understanding of their own and others’ experiences (7). Moreover, it is effective in reducing suicidal behavior, anger, and impulsivity, as well as in enhancing psychosocial functioning (8). However, several challenges may also be encountered during psychodynamic psychotherapy, such as intense transference and countertransference, which can hinder the therapist-patient relationship. Therefore, understanding and managing the transference-countertransference dynamics in therapy present central strategies in conducting psychodynamic psychotherapy for BPD to prevent such reactions from disrupting the therapeutic relationship and subsequently improve therapeutic benefits. Therefore, to be psychodynamic psychotherapists for individuals with BPD, therapists require self-awareness, perseverance, and neutrality to survive intense transference-countertransference reactions and strike a sound balance between empathy and detachedness (9, 10).

The Indonesian College of Psychiatry has defined the competency of performing psychodynamic psychotherapy as a general competency that must be achieved by psychiatry residents who become full-fledged therapists once they graduate and can perform psychodynamic psychotherapy for BPD (11). Therapist competence has been conceptualized as the level of the therapist’s knowledge and skill to implement a treatment to an acceptable standard (12, 13). The need to evaluate therapists’ competence in psychotherapy has been widely recognized (14). Psychodynamic psychotherapists’ competencies are essential as higher levels of competence lead to better patient outcomes, such as clinical symptoms and social functioning (15, 16). However, deficiencies or misapplication of these techniques are related to negative therapy outcomes (17). Assessment of psychotherapeutic competence could be achieved through knowledge tests, evaluation of patient outcomes, patient feedback, or evaluation of treatment sessions using evaluation scales (12, 15, 18). Structured evaluation scales facilitate a systematic assessment of competence (19). Using a competency assessment scale also enables the provision of performance-based feedback for trainee psychiatrists (20). Most of these scales are constructed as a rating scale to be used by supervisors based on their observation of the psychotherapists’ performance and are developed based on expert consensus or the authors’ expertise (13, 21). Although several standardized psychotherapy competency assessment scales have been developed for psychotherapists, these are primarily used in general practice, such as the Cognitive Therapy Scale-Revised or the Supervisor’s Evaluation Scale (15, 22–24). Several competency assessment scales have been developed to assess psychotherapeutic competency based on distinct mental disorders. Currently, standardized competency assessment scales for evaluating psychotherapists’ competency in conducting psychodynamic psychotherapy for BPD are limited.

Moreover, limited psychotherapy competence scales exist that implement the assessment of the three domains of learning, namely the cognitive, affective, and psychomotor domains, as defined by Bloom in the Taxonomy of Educational Objectives. Each learning domain comprises hierarchical categories of learning objectives ranging from the simplest to the most complex, which can be applied to assessment scales (25). The application of Bloom’s domains of learning in medical education has helped establish a model of clinical competence as a combination of intellectual knowledge, attitudes or awareness of values, and psychomotor abilities (26). Assessments should involve all domains of learning and specify the learning objectives that must be achieved so that the competence being assessed can be comprehensively measured (27). Additionally, a method of assessment should possess good psychometric properties, such as validity and reliability, to ensure its ability to correctly measure the intended performance, and the results should be consistent across different instances and raters (15, 28). Therefore, it is essential to develop comprehensive scales to assess cognitive, affective, and psychomotor competency (CS-CAPC) in psychodynamic psychotherapy for BPD to determine the competency of psychodynamic psychotherapists who conduct psychodynamic psychotherapy in individuals with BPD. The CS-CAPC comprises two scales. The first scale, the psychodynamic formulation competency assessment scale (PF-CAS), was developed to assess the skill of creating a psychodynamic formulation by evaluating the therapist’s written psychodynamic formulation. The second scale, the practical competency assessment scale (PC-CAS) in psychodynamic psychotherapy for BPD, was developed to assess therapists’ skills in performing psychodynamic psychotherapy for patients with BPD in real-life clinical settings by assessing a video recording of a psychotherapy session or by direct observation of a session. The PF-CAS was designed for cognitive and affective domain evaluations, whereas the PC-CAS in psychodynamic psychotherapy for BPD was designed to assess the cognitive, affective, and psychomotor domains.

2 Materials and method

2.1 Study design

The development of the CS-CAPC comprises two phases: (1) data collection and (2) measurement of the validity of the scales. For data collection, an initial item list was developed based on a literature review and Delphi survey, and a panel discussion was held to explore experts’ and stakeholders’ perceptions of the required characteristics and components of the scales. The validity of the instruments was tested using content validity testing. This study was conducted between September 2022 and August 2023 using Zoom’s online discussion platform and distributing online forms via email.

2.2 Ethics statement

Ethical approval was obtained from the Health Research Ethics Committee of the Faculty of Medicine, Universitas Indonesia (protocol number: 22-10-1201; KET-1074 UN2.F1/ETIK/PPM.00.02/2022).

2.3 Participants and procedure

Psychotherapy experts and stakeholders in psychiatric education were invited to provide their perceptions and feedback on these scales. We utilized criterion-based purposive sampling to recruit participants based on their experience and expertise in relevant fields. Each part of the study involved a different set of participants. All participants were invited to participate in the study and were informed of its aims and procedures. Written informed consent was obtained from all participants before their enrolment in the study.

2.3.1 Delphi survey

The Delphi technique is a common method used to achieve consensus among a panel of experts or stakeholder groups. It is performed by conducting iterative rounds of surveys until a consensus is reached (29). Seven psychiatrists from faculty members of psychiatry residency-program institutions in Indonesia and council members of the Psychotherapy Section of the Indonesian Psychiatric Association were invited to participate in the survey.

For the Delphi survey, we developed the initial item lists for the PF-CAS and PC-CAS in psychodynamic psychotherapy for BPD based on a literature review. These scales were designed and intended for use in the Indonesian language. Each item on the list was accompanied by a short description of the level of cognitive, affective, or psychomotor learning to be achieved. There were seven items in the PF-CAS and 12 items in the PC-CAS in psychodynamic psychotherapy for BPD. An online Delphi survey was conducted by sending initial item lists to seven experts via email. In each round, the experts rated the items on a 4-point Likert scale of 1 (strongly disagree), 2 (disagree), 3 (agree), and 4 (strongly agree). Moreover, the experts were free to provide written suggestions, such as revisions, deletions, or additions to the items. The principal investigators summarized the scores and feedback and returned the results to the experts in the next round until a consensus was reached. Descriptive statistics were used to calculate the experts’ mean scores. A consensus was defined as a mean score of ≥ 3.5 (30).

2.3.2 Panel discussion

Following the Delphi method, finalized item drafts were developed into two scales in the form of evaluation rubrics. We designed the rubric components defined by Burghart and Panettieri (31): task description, scale of achievement, dimensions, and description of dimensions. The names of the scales represent task descriptions, whereas the evaluation items represent the dimensions of the task. Each item was quantitatively scored on a scale from 0 to 3, with 0 indicating the lowest and 3 indicating the highest score. Each score on the scale has a description. All the item scores were totaled and converted into numeric scores of 0–100 to determine the final score.

The panel discussion was conducted as a focus group discussion via the online discussion platform, Zoom Meetings, to reach an agreement on all scale items and descriptions. The panel discussion included three experts in psychotherapy from members of the Psychotherapy Section of the Indonesian Psychiatric Association. Inclusion criteria were (1) a consultant in psychotherapy, and (2) a faculty member of a psychiatric education institution or having experience in providing psychotherapy education through workshops. Exclusion criteria were (1) refusing to participate in the study, and (2) are no longer active as a clinician. There were one male expert and two female experts. The mean age was 56.7 years old. The experts had experience in psychotherapy ranging from 10 to 31 years, with a mean of 23 years. Data from the discussions were analyzed using thematic analysis. The resulting scales were then tested for content validity.

2.3.3 Content validity testing

A content validity test for the CS-CAPC in psychodynamic psychotherapy for BPD was conducted to investigate each item’s relevance. We distributed the scales along with a content validation questionnaire via email to the participants, who then rated each item’s significance on a 4-point Likert scale: 1 (very irrelevant/VI); 2 (not relevant/NR); 3 (relevant/R); and 4 (very relevant/VR). Participants could also provide written feedback on the items.

Content validity was determined by calculating the item-level content validity index (I-CVI) for each scale item and the scale’s average scale-level (S-CVI/Ave) values. The I-CVI was calculated by dividing the number of participants who allocated a rating of 3 or 4 by the total number of participants. The S-CVI/Ave was calculated by averaging the I-CVI for all scale items. An I-CVI value of > 0.79 and an S-CVI/Ave value of > 0.78 were deemed acceptable (32).

Nine faculty psychiatrists who teach psychotherapy in their respective residency programs were invited to participate in the content validity testing. We employed stratified purposive sampling to recruit one psychotherapy teaching staff member from each of the nine psychiatric residency-program institution centers in Indonesia.

Two males and seven females were selected. Regarding age (years), one faculty psychiatrist was between 30 and 40, five were between 40 and 50, one was between 50 and 60, and two were between 60 and 70. Two were consultants in child and adolescent psychiatry, two were consultants in consultation-liaison psychiatry, and five were general psychiatrists. All had experience in teaching psychotherapy, with years of teaching experience ranging from 2 to 23 years and a mean of 14.5 years.

3 Results

3.1 Delphi survey

Figure 1 illustrates the flow of the Delphi survey. Seven experts participated in the Delphi survey, with a mean age of 59.5 years and experience ranging from 5 to 31 years. Demographic characteristics of the experts are shown in Table 1. We conducted two rounds of the Delphi analysis. In the first round, all the items in PF-CAS achieved the consensus criteria of a mean score ≥3.5 and were accepted. For the PC-CAS in psychodynamic psychotherapy for BPD, one item (Item 2), “Showing a warm attitude to the patient,” obtained a mean score of 3.28, lower than the predefined criteria of consensus. The item was revised according to the written feedback and included in the second Delphi round: “Paying attention to the patient without judgment and prejudice.” The remaining items on PC-CAS in psychodynamic psychotherapy for BPD reached consensus with a mean score in the range 3.57–3.85. However, item 4, “Maintaining good eye contact throughout the therapy session (for at least 85% of the session),” received revision suggestions from two of the experts. Thus, we included the item in the second Delphi round. Therefore, the second Delphi round was held for the PC-CAS in psychodynamic psychotherapy for BPD items which still needed revisions.

Figure 1

Figure 1. Delphi survey flowchart.

Table 1

Table 1. Demographic characteristics of Delphi survey participants (N=7).

The new version of the PC-CAS in psychodynamic psychotherapy for BPD, comprising the revised items, was sent back to the experts for the second Delphi round. A consensus was reached on all items, with a mean score of 3.71. Written feedback in the second round was used to develop descriptions of the item-scoring criteria for the study’s next step. Table 2 shows the results of Delphi rounds 1 and 2. The resulting draft of the scales comprised 7 items on the PF-CAS and 12 items on the PC-CAS in psychodynamic psychotherapy for BPD (Table 3).

Table 2

Table 2. Delphi Round 1 and 2 Results.

Table 3

Table 3. The resulting draft of the scales from the Delphi survey.

3.2 Panel discussion

The items agreed upon in the Delphi survey were developed into two scales. Each item was scored on a scale of 0–3, with descriptions for each score and a description of the level of cognitive, affective, and psychomotor learning assigned to each item. The draft scales were then subjected to a panel discussion among the three experts.

Based on the panel discussion, the number of items in the new iteration of the PF-CAS changed from seven to six. Changes were made to Items 5 and 6, which were combined into one item. The changes were as follows: “Explaining the management plan for the potential resistance, transference, and countertransference that might occur in the therapy sessions” and “Explaining the plan and goals of the psychotherapy” were combined into “Explaining the goals of the psychotherapy and the management plan for the potential resistance, transference, and countertransference that might occur in the therapy sessions.” Accordingly, the description of Score 3 criteria in Item 5 was changed from “Clearly describes the management plan for the potential resistance, transference, and countertransference that might occur in the therapy sessions” to “Clearly describes the goals of the psychotherapy and the management plan for the potential resistance, transference, and countertransference that might occur in the therapy sessions.”

There are 12 items on the PC-CAS in psychodynamic psychotherapy for BPD. The panel discussion resulted in several revisions to the draft, namely, those to Items 6, 8, 10, and 11. The change to Item 6 was in the description of Score 1, from “The therapist rarely paraphrases” to “The therapist very rarely paraphrases or paraphrases too often.” For Item 8, a minor grammatical change was made to the description of Score 1 to increase the clarity of the sentence. In Item 10, the change in Score 3 was from “The therapist interprets and discusses the patient’s transference so that the patient understands themselves and their problems better” to “The therapist interprets and discusses the patient’s transference so that the patient gradually understands themselves and their problems better.” The final change in the scale was on Item 11 for the description of Score 1, from “The therapist appears to try managing their feelings and responds to the patient, but the response is inappropriate, and they seem to be unable to navigate the session,” to “The therapist appears to recognize their feelings and try to manage them, responds to the patient but the response is inappropriate, and they seem to be unable to navigate the session.”

3.3 Content validity testing

Both scales are deemed valid based on an S-CVI/Ave value of 0.981 for the PF-CAS and an S-CVI/Ave value of 1.00 for the PC-CAS in psychodynamic psychotherapy for BPD. The I-CVI values for all six items in the PF-CAS ranged from 0.889 to 1.00. All items received a score of 3 or 4, except for Item 5, which received eight scores of 3 or 4 and one of 2. Items 1, 4, and 6 received minor revisions from the participants. The scoring scale on Item 1 was modified for better specificity of the scale. Items 4 and 6 were revised for better wording.

The I-CVI for all twelve items in the PC-CAS was 1.00. All items attained a score of 3 or 4. Modifications were made to the scoring scale descriptions of Item 12 for better specificity of the scoring scale based on feedback from the three participants. Table 4 shows the results of the content validity testing. Examples of the feedback and revisions can be seen in Table 5.

Table 4

Table 4. Content validity testing results.

Table 5

Table 5. Examples of participants’ content validity feedback for the scales.

The scales were revised and finalized, the finalized scales are shown in Tables 6 and 7.

Table 6

Table 6. Psychodynamic formulation competency assessment scale.

Table 7

Table 7. Practical competency assessment scale in psychodynamic psychotherapy for BPD.

4 Discussion

This study described the psychometric properties of comprehensive assessment scales for the competency of psychodynamic psychotherapy for BPD among individuals who were trained as psychodynamic psychotherapists for BPD patients. The decision to design the scales in scoring rubrics was considered for the utility of rubrics in performance-based assessment. A scoring rubric objectively evaluates complex skills or behaviors by breaking them down into observable criteria (33, 34).

The CS-CAPC in psychodynamic psychotherapy for BPD was developed through the sequential activities of a Delphi survey, panel discussion, and validity testing. The scales in CS-CAPC in psychodynamic psychotherapy for BPD indicated satisfactory content validity, with S-CVI/Ave = 0.981 for the PF-CAS and S-CVI/Ave = 1.00 for the PC-CAS. The data collected during content validation indicated that all scale items were relevant for assessing the skills of psychodynamic formulation writing and conducting psychodynamic psychotherapy for BPD in real-life clinical settings. Most qualitative feedback was centered on creating a clear distinction between all the scores on the scale, allowing us to ensure an accurate grading of the assessed competency. Every revision of the scale was made based on participant feedback. One of the strengths of this study was the recruitment of a diverse sample of psychotherapy experts, psychiatrists experienced in psychotherapy, and teaching staff psychiatrists from nine psychiatry residency program institutions in Indonesia, allowing them to provide written feedback for the scale items during all activities. This helped us accumulate opinions and perspectives on the scales from the major stakeholders directly involved in learning situations across various institutions in Indonesia.

The CS-CAPC was designed to evaluate the competency of psychodynamic psychotherapy for BPD, as measured in the cognitive, affective, and psychomotor domains. This characteristic is among the novelties of our scale, as no previous scales for the competency of psychodynamic psychotherapy, let alone psychotherapy, defined the cognitive, affective, and psychomotor domains present in the task. The PF-CAS was used to evaluate the level of therapists’ competency in the cognitive and affective domains. The psychodynamic formulation is a tentative hypothesis that contains a succinct conceptualization of a patient’s clinical picture and guides the treatment plan (35). Our assignment of the cognitive and affective domains to the competency of writing a psychodynamic formulation was positively aligned with how creating a psychodynamic formulation requires the therapist’s willingness to delve into the patient’s internal world to discern central conflicts and themes in their lives (36). Mace and Binyon (37) stated that, in addition to gathering information from questioning, therapists might need to reflect on the feelings they experienced when interacting with a patient to infer the patient’s characteristic style of interpersonal relationships. These requirements correspond to the affective domain of learning and cognitive domain of understanding the theoretical framework of the psychoanalytic theories (20).

For the PC-CAS in psychodynamic psychotherapy for BPD, which assesses the cognitive, affective, and psychomotor domains, Ackerman and Hilsenroth (38) previously discussed how performing psychotherapy required a combination of cognitive, affective, and interpersonal skills. Several non-verbal elements are ubiquitous in psychotherapy, including eye contact, aligning body gestures per the patient’s emotional state, and voice and interruption behaviors (39, 40). This supports our assignment of the psychomotor domain to all items of the PC-CAS in psychodynamic psychotherapy for BPD.

The final version of the PF-CAS contained six items. The items on our scale reflect the structured components of a psychodynamic formulation as delineated in the literature. Sperry (41) defined three components of psychodynamic formulation: (1) a description of the patient’s clinical picture and stressors; (2) an etiological rationale for the factors contributing to the patient’s clinical picture; and (3) a formulation of treatment and prognosis based on the first two components. Similarly, a format proposed by Perry (42) and later updated by Summers (43) outlined the four essential components of a psychodynamic formulation: (1) summary of the patient’s problems; (2) description of the non-dynamic factors; (3) psychodynamic explanation of the patient’s central conflicts; and (4) prediction of the course of the therapy. The first item on our scale, “describing the patient’s condition and stressors,” summarizes the patient’s clinical picture and related stressors. The second item, “formulating the biological, intrapsychic, and sociocultural factors that influence the patient’s condition,” corresponds with describing the factors that contributed to the patient’s condition. The third item, “determining the suitable psychoanalytic theory to explain the patient’s condition,” embodies the formulation of the patient’s central conflicts using one or more psychodynamic theories. The fourth, fifth, and sixth items, “describing the potential resistance, transference, and countertransference that might occur in the therapy sessions,” “explaining the goals of the psychotherapy and the management plan for the potential resistance, transference, and countertransference that might occur in the therapy sessions,” and “explaining the patient’s prognosis in the course of the psychotherapy,” reflect the formulation of treatment and prognosis.

Our findings on the PF-CAS are consistent with those in previous studies that attempted to develop a method to assess written psychodynamic formulations. The case formulation content coding method (CFCCM) designed by Eells et al. (44) was used to evaluate the completeness and quality of the case formulation based on four content areas: (1) symptoms and problems; (2) the patient’s precipitating stressors; (3) predisposing life events; and (4) an explanation of the patient’s current difficulties by linking the three previous categories. The CFCCM rates the quality of the formulation as a whole and in the three dimensions of complexity, degree of analytic inference, and precision of language on a 5-point scale (44). Our rating scale contains a combination of the dimensions in the CFCCM’s rating scale, with the highest level of 3 representing the most comprehensive, analytical, and easily understood explanation of each component. Other models have focused on assessing the completeness of a patient’s psychodynamic conceptualization, such as Perry et al.’s (45) idiographic conflict formulation method and Curtis et al.’s (46) plan diagnosis method. The PF-CAS assesses the components of the patient’s psychodynamic conceptualization, similar to these existing methods, and it adds items that evaluate the hypothetical course of the therapy and treatment plan.

The PC-CAS in psychodynamic psychotherapy for BPD comprises 12 items: (1) beginning the session; (2) paying attention to the patient without judgment and prejudice; (3) conducting conversations that aim to build rapport with the patient; (4) maintaining good eye contact throughout the therapy session for most of the time; (5) aligning the gestures and posture per the patient’s state during the therapy session; (6) paraphrasing the patient’s statements to show that the therapist is empathetic and understanding toward the patient’s feelings; (7) using appropriate PP interventions per the patient’s state; (8) managing the patient’s resistance in the session; (9) responding adequately to the changes in the patient’s emotional states; (10) managing the patient’s transference; (11) managing countertransference reactions; and (12) summarizing and closing the session. Our findings agree with previous evaluation scales for competency in psychodynamic psychotherapy (20, 22). The rating scale developed by Winer and Mostert (20) evaluates the following items: establishing a therapeutic situation; facilitating the formation of a therapeutic alliance; recognizing the therapist’s own emotional reactions; experiencing the patient’s feelings while maintaining objectivity; communicating an empathic understanding in a way that enables the patient to feel understood; detecting multiple meanings in the patient’s communication; making interpretations; and formulating a psychodynamic explanation for the patient. The 29-item supervisor’s evaluation scale contains items that assess residents’ capacity to establish a working alliance of empathy, recognize and interpret transference, deal with resistance and countertransference, and close the session (22). The new item proposed in the PC-CAS in psychodynamic psychotherapy for BPD assesses a therapist’s ability to respond to a patient’s emotional changes. Maroda (47) and Kernberg (48) have stated that a therapist’s capacity to respond to a patient’s emotions plays a vital role in the psychodynamic psychotherapy of borderline patients, as emotional dysregulation causes these patients to exhibit frequent bouts of “affect storms.”

The PC-CAS in psychodynamic psychotherapy for BPD also emphasized the management of transference and countertransference in the psychodynamic psychotherapy of BPD. Transference interpretations may be viewed as “high-risk, high-gain” interventions in the psychotherapy of borderline patients, as these patients appear to be more vulnerable and easily overwhelmed by transference interpretations (9, 49). Transference interpretations in the psychodynamic psychotherapy of borderline patients must be performed in a timely and appropriate yet still evocative manner, to propel the process of psychotherapy (9, 20, 49). These features are consistent with the emphasis on timing and directed elaboration when managing transference in the PC-CAS in psychodynamic psychotherapy for BPD. Working with borderline patients is known to elicit a wide array of strong countertransference reactions within the therapists, including rage and hatred, guilt, inadequacy, anxiety, and parental feelings (9, 49). A meta-analysis by Hayes et al. (50) found that the management of countertransference was related to better therapeutic outcomes. However, the scope of this meta-analysis was not restricted to psychodynamic psychotherapy and borderline patients. The countertransference factors inventory (CFI) measures therapists’ countertransference management behavior by assessing five attributes related to countertransference management: self-insight, self-integration, anxiety management, empathy, and conceptualizing ability (51). The characteristics of self-insight, self-integration, anxiety management, and conceptualizing ability are reflected within the highest score of Item 11 on the PC-CAS in psychodynamic psychotherapy for BPD. Meanwhile, the empathy attribute is reflected in Item 6 on the PC-CAS in psychodynamic psychotherapy for BPD.

Our study had several strengths and limitations. The first strength is the robust data collection from a diverse sample of psychotherapy experts, psychiatrists experienced in psychotherapy, and teaching staff psychiatrists in Indonesia. The number of samples in the Delphi method, panel discussion, and content validity testing fulfilled the minimum number of participants for the respective activities (31, 52–54). Another strength of this study lies in the comprehensive steps taken in developing and validating the scales, including collecting qualitative feedback. The limitation of the study is the small number of experts in the panel discussion due to the limited number of psychotherapy consultants in Indonesia. The scales demonstrated potential utility in assessing the specific competency of psychodynamic psychotherapy for BPD and could be used in both learning and clinical situations. Future studies should investigate the application of these scales in education and clinical environments. The scales could be used for psychotherapy training or monitoring purposes and the results could be analyzed in comparison with other measures of psychotherapeutic competence, such as patient outcome.

5 Conclusion

The CS-CAPC in psychodynamic psychotherapy for BPD was designed to evaluate the cognitive, affective, and psychomotor domains of competency in psychodynamic psychotherapy for BPD, and it was found to have satisfactory content validity. The first scale in the CS-CAPC in psychodynamic psychotherapy for BPD, PF-CAS, could be used to assess the quality of written psychodynamic formulation. Meanwhile, the second scale, PC-CAS, assesses the practice of psychodynamic psychotherapy for BPD through direct or indirect observation via video recordings of the therapists’ psychotherapy sessions. The scale serves as a tool for a more objective and systematic assessment of the competency of psychodynamic psychotherapy for BPD, which could aid the didactic practice for psychodynamic psychotherapy for BPD among trainee therapists, such as psychiatry residents.

Data availability statement

The datasets presented in this article are not readily available because the participants did not agree to publicly share the data they provided for the study. Requests to access the datasets should be directed to cHRybjEwMTBAeWFob28uY29t.

Ethics statement

The studies involving humans were approved by Health Research Ethics Committee, Faculty of Medicine, Universitas Indonesia. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

PL: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing – original draft, Writing – review & editing, Investigation, Supervision. TW: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Formal analysis, Supervision. DS: Conceptualization, Formal analysis, Methodology, Supervision, Writing – review & editing. SM: Conceptualization, Methodology, Supervision, Writing – review & editing. SE: Writing – review & editing. LS: Writing – review & editing. TS: Writing – review & editing. AK: Formal analysis, Methodology, Writing – review & editing. RN: Data curation, Formal analysis, Investigation, Writing – review & editing. HR: Data curation, Formal analysis, Investigation, Writing – review & editing. KG: Data curation, Formal analysis, Investigation, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

We would like to thank all study participants for their rich insights and perspectives on our scales. We would also like to thank Prof. Ari Fahrial Syam, MD, the Dean of the Faculty of Medicine, Universitas Indonesia, Prof. Suhendro, MD, and Prof. Harrina Rahardjo, MD, from the Doctoral Program in Medical Science of the Faculty of Medicine, Universitas Indonesia, for their guidance and support. We also thank the members of the Psychotherapy Section of the Indonesian Psychiatric Association for their contributions to psychotherapy in Indonesia.

Conflict of interest

The authors declare that this study was conducted without any commercial or financial relationships that could be construed as potential conflicts of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Merced M. The beginning psychotherapist and borderline personality disorder: basic treatment principles and clinical foci. Apt Am J Psychother. (2015) 69:241–68. doi: 10.1176/appi.psychotherapy.2015.69.3.241

Crossref Full Text | Google Scholar

2. Winsper C, Bilgin A, Thompson A, Marwaha S, Chanen AM, Singh SP, et al. The prevalence of personality disorders in the community: a global systematic review and meta-analysis. Br J Psychiatry. (2020) 216:69–78. doi: 10.1192/bjp.2019.166

PubMed Abstract | Crossref Full Text | Google Scholar

3. Kernberg OF, Michels R. Borderline personality disorder. Am J Psychiatry. (2009) 166:505–8. doi: 10.1176/appi.ajp.2009.09020263

PubMed Abstract | Crossref Full Text | Google Scholar

4. Chapman J, Jamil R, Fleisher C. Borderline personality disorder. Treasure Island: StatPearls Publishing (2022).

Google Scholar

5. Dr. Cipto Mangunkusumo National Central General Hospital. Patient Registry of the Psychiatric Outpatient Clinic and Inpatient Ward. Jakarta: Dr. Cipto Mangunkusumo National Central General Hospital (2022).

Google Scholar

6. Biskin RS, Paris J. Management of borderline personality disorder. CMAJ. (2012) 184:1897–902. doi: 10.1503/cmaj.112055

PubMed Abstract | Crossref Full Text | Google Scholar

7. Gonzalez-Torres MA. Psychodynamic psychotherapies for borderline personality disorders. Current developments and challenges ahead. BJPsych Int. (2018) 15:12–4. doi: 10.1192/bji.2017.7

PubMed Abstract | Crossref Full Text | Google Scholar

8. Zanarini MC. Psychotherapy of borderline personality disorder. Acta Psychiatr Scand. (2009) 120:373–7. doi: 10.1111/j.1600-0447.2009.01448.x

PubMed Abstract | Crossref Full Text | Google Scholar

9. Bhola P, Mehrotra K. Associations between countertransference reactions towards patients with borderline personality disorder and therapist experience levels and mentalization ability. Trends Psychiatry Psychother. (2021) 43:116–25. doi: 10.47626/2237-6089-2020-0025

PubMed Abstract | Crossref Full Text | Google Scholar

10. Arthur AR. Psychodynamic counselling for the borderline personality disordered client: a case study. Psychodyn Couns. (2000) 6:31–48. doi: 10.1080/135333300362846

Crossref Full Text | Google Scholar

11. Indonesian College of Psychiatry. Curriculum for Psychiatry Residency Program. Jakarta: Indonesian College of Psychiatry (2012).

Google Scholar

12. Liston EH, Yager J, Strauss GD. Assessment of psychotherapy skills: the problem of interrater agreement. Am J Psychiatry. (1981) 138:1069–74. doi: 10.1176/ajp.138.8.1069

PubMed Abstract | Crossref Full Text | Google Scholar

13. Fairburn CG, Cooper Z. Therapist competence, therapy quality, and therapist training. Behav Res Ther. (2011) 49:373–8. doi: 10.1016/j.brat.2011.03.005

PubMed Abstract | Crossref Full Text | Google Scholar

14. Barber JP, Sharpless BA, Klostermann S, McCarthy KS. Assessing intervention competence and its relation to therapy outcome: a selected review derived from the outcome literature. Prof Psychol Res Pr. (2007) 38:493–500. doi: 10.1037/0735-7028.38.5.493

Crossref Full Text | Google Scholar

15. Barlow I, Brown RJ. A systematic review of measures of therapist competence in psychodynamic, interpersonal, and/or relational models. Psychol Psychother. (2020) 93:408–27. doi: 10.1111/papt.12231

PubMed Abstract | Crossref Full Text | Google Scholar

16. Davidson K, Scott J, Schmidt U, Tata P, Thornton S, Tyrer P. Therapist competence and clinical outcome in the Prevention of parasuicide by Manual Assisted Cognitive Behaviour Therapy trial: the POPMACT study. Psychol Med. (2004) 34:855–63. doi: 10.1017/s0033291703001855

PubMed Abstract | Crossref Full Text | Google Scholar

17. Hadley SW, Strupp HH. Contemporary views of negative effects in psychotherapy. an integrated account. Arch Gen Psychiatry. (1976) 33:1291–302. doi: 10.1001/archpsyc.1976.01770110019001

PubMed Abstract | Crossref Full Text | Google Scholar

18. Yager J, Bienenfeld D. How competent are we to assess psychotherapeutic competence in psychiatric residents? Acad Psychiatry. (2003) 27:174–81. doi: 10.1176/appi.ap.27.3.174

PubMed Abstract | Crossref Full Text | Google Scholar

19. Easden MH, Fletcher RB. Therapist competence in case conceptualization and outcome in CBT for depression. Psychother Res. (2020) 30:151–69. doi: 10.1080/10503307.2018.1540895

PubMed Abstract | Crossref Full Text | Google Scholar

20. Winer JA, Mostert M. Evaluation of residents’ dynamic psychotherapy skills. Acad Psychiatry. (1988) 12:329–37. doi: 10.1007/BF03399994

Crossref Full Text | Google Scholar

21. Gonsalvez CJ, Crowe TP. Evaluation of psychology practitioner competence in clinical supervision. Am J Psychother. (2014) 68:177–93. doi: 10.1176/appi.psychotherapy.2014.68.2.177

PubMed Abstract | Crossref Full Text | Google Scholar

22. Buckley P, Conte HR, Plutchik R, Karasu TB, Wild KV. Learning dynamic psychotherapy: a longitudinal study. Am J Psychiatry. (1982) 139:1607–10. doi: 10.1176/ajp.139.12.1607

PubMed Abstract | Crossref Full Text | Google Scholar

23. Flemotomos N, Martinez VR, Chen Z, Singla K, Ardulov V, Peri R, et al. Automated evaluation of psychotherapy skills using speech and language technologies. Behav Res Methods. (2022) 54:690–711. doi: 10.3758/s13428-021-01623-4

PubMed Abstract | Crossref Full Text | Google Scholar

24. Blackburn IM, James IA, Milne DL, Baker C, Standart S, Garland A, et al. The revised cognitive therapy scale (CTS-R): psychometric properties. Behav Cognit Psychother. (2001) 29:431–46. doi: 10.1017/S1352465801004040

Crossref Full Text | Google Scholar

25. Bilon E. Using Bloom’s taxonomy to write effective learning objectives: the ABCDs of writing learning objectives : a basic guide. La Vergne: Edmund Bilon (2019).

Google Scholar

26. Nascimento JDSG, Siqueira TV, Oliveira JLG, Alves MG, Regino DDSG, Dalri MCB. Development of clinical competence in nursing in simulation: the perspective of Bloom’s taxonomy. Rev Bras Enferm. (2021) 74:e20200135. doi: 10.1590/0034-7167-2020-0135

PubMed Abstract | Crossref Full Text | Google Scholar

27. Shaw LM. Effective assessment of trainees. Obstetric Gynaecologis. (2004) 6:171–7. doi: 10.1576/toag.6.3.171.27001

Crossref Full Text | Google Scholar

28. Jonsson A, Svingby G. The use of scoring rubrics: reliability, validity and educational consequences. Educ Res Rev. (2007) 2:130–44. doi: 10.1016/j.edurev.2007.05.002

Crossref Full Text | Google Scholar

29. Biggane AM, Williamson PR, Ravaud P, Young B. Participating in core outcome set development via Delphi surveys: qualitative interviews provide pointers to inform guidance. BMJ (Open). (2019) 9:e032338. doi: 10.1136/bmjopen-2019-032338

PubMed Abstract | Crossref Full Text | Google Scholar

30. Rankin G, Rushton A, Olver P, Moore A. Chartered Society of Physiotherapy’s identification of national research priorities for physiotherapy using a modified Delphi technique. Physiotherapy. (2012) 98:260–72. doi: 10.1016/j.physio.2012.03.002

PubMed Abstract | Crossref Full Text | Google Scholar

31. Burghart G, Panettieri RC. Faculty guide to rubrics. Radiol Technol. (2009) 80:266–8.

PubMed Abstract | Google Scholar

32. Department of Medical Education, School of Medical Sciences, Universiti Sains Malaysia, MALAYSIA, Yusoff MSB. ABC content validation content validity index calculation. EIMJ. (2019) 11:49–54. doi: 10.21315/eimj2019.11.2.6

Crossref Full Text | Google Scholar

33. Ramazanzadeh N, Ghahramanian A, Zamanzadeh V, Valizadeh L, Ghaffarifar S. Development and psychometric testing of a clinical reasoning rubric based on the nursing process. BMC Med Educ. (2023) 23:98. doi: 10.1186/s12909-023-04060-3

PubMed Abstract | Crossref Full Text | Google Scholar

34. Renjith V, George A, Renu G, D’Souza P. Rubrics in nursing education. Int J Adv Res. (2015) 3:423–8.

Google Scholar

35. Gabbard GO. Textbook of Psychotherapeutic Treatments. 1st ed. Washington, DC: American Psychiatric Publishing (2009). p. 876.

Google Scholar

36. Böhmer MW. Dynamic psychiatry and the psychodynamic formulation. Afr J Psychiatry (Johannesburg). (2011) 14:273–7. doi: 10.4314/ajpsy.v14i4.3

Crossref Full Text | Google Scholar

37. Mace C, Binyon S. Teaching psychodynamic formulation to psychiatric trainees: Part 1: Basics of formulation. Adv Psychiatr Treat. (2005) 11:416–23. doi: 10.1192/apt.11.6.416

Crossref Full Text | Google Scholar

38. Ackerman SJ, Hilsenroth MJ. A review of therapist characteristics and techniques positively impacting the therapeutic alliance. . Clin Psychol Rev. (2003) 23:1–33. doi: 10.1016/s0272-7358(02)00146-0

PubMed Abstract | Crossref Full Text | Google Scholar

39. Foley GN, Gentile JP. Nonverbal communication in psychotherapy. Psychiatry (Edgmont). (2010) 7:38–44.

PubMed Abstract | Google Scholar

40. Del Giacco L, Anguera MT, Salcuni S. The action of verbal and nonverbal communication in the therapeutic alliance construction: A mixed methods approach to assess the initial interactions with depressed patients. Front Psychol. (2020) 11:1351. doi: 10.3389/fpsyg.2020.01351

PubMed Abstract | Crossref Full Text | Google Scholar

41. Sperry L. Demystifying the psychiatric case formulation. Jefferson J Psychiatry. (1992) 10:12–9. doi: 10.29046/JJP.010.2.002

Crossref Full Text | Google Scholar

42. Perry S, Cooper AM, Michels R. The psychodynamic formulation: its purpose, structure, and clinical application. Am J Psychiatry. (1987) 144:543–50. doi: 10.1176/ajp.144.5.543

PubMed Abstract | Crossref Full Text | Google Scholar

43. Summers RF. The psychodynamic formulation updated. Am J Psychother. (2003) 57:39–51. doi: 10.1176/appi.psychotherapy.2003.57.1.39

PubMed Abstract | Crossref Full Text | Google Scholar

44. Eells TD, Kendjelic EM, Lucas CP. What’s in a case formulation? Development and use of a content coding manual. J Psychother Pract Res. (1998) 7:144–53.

PubMed Abstract | Google Scholar

45. Perry JC, Knoll M, Tran V. Motives, defences, and conflicts in the dynamic formulation for psychodynamic psychotherapy using the idiographic conflict formulation method. In: Case formulation for personality disorders. Amsterdam: Elsevier (2019). p. 203–24. Available at: https://linkinghub.elsevier.com/retrieve/pii/B9780128135211000114.

Google Scholar

46. Curtis JT, Silberschatz G, Sampson H, Weiss J, Rosenberg SE. Developing reliable psychodynamic case formulations: an illustration of the plan diagnosis method. Psychother Theor Res Pract Train. (1988) 25:256–65. doi: 10.1037/h0085340

Crossref Full Text | Google Scholar

47. Maroda KJ. Psychodynamic Techniques – Working with Emotion in the Therapeutic Relation. New York: Guilford Press (2012).

Google Scholar

48. Kernberg OF. The management of affect storms in the psychoanalytic psychotherapy of borderline patients. J Am Psychoanal Assoc. (2003) 51:517–45. doi: 10.1177/00030651030510021101

PubMed Abstract | Crossref Full Text | Google Scholar

49. Gabbard GO, Horwitz L, Allen JG, Frieswyk S, Newsom G, Colson DB, et al. Transference interpretation in the psychotherapy of borderline patients: a high-risk, high-gain phenomenon. Harv Rev Psychiatry. (1994) 2:59–69. doi: 10.3109/10673229409017119

PubMed Abstract | Crossref Full Text | Google Scholar

50. Hayes JA, Gelso CJ, Goldberg S, Kivlighan DM. Countertransference management and effective psychotherapy: meta-analytic findings. Psychother (Chic). (2018) 55:496–507. doi: 10.1037/pst0000189

Crossref Full Text | Google Scholar

51. Gelso CJ, Latts MG, Gomez MJ, Fassinger RE. Countertransference management and therapy outcome: an initial evaluation. J Clin Psychol. (2002) 58:861–7. doi: 10.1002/jclp.2010

PubMed Abstract | Crossref Full Text | Google Scholar

52. Mahajan V, Linstone HA, Turoff M. The Delphi method: techniques and applications. J Mark Res. (1976) 13:317. doi: 10.2307/3150755

Crossref Full Text | Google Scholar

53. Giannarou L, Zervas E. Using Delphi technique to build consensus in practice. Int J Bus Sci Appl Manag. (2014) 9:65–82. doi: 10.69864/ijbsam.9-2.106

Crossref Full Text | Google Scholar

54. Bojke L, Soares M, Claxton K, Colson A, Fox A, Jackson C, et al. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technol Assess. (2021) 25:1–124. doi: 10.3310/hta25370

PubMed Abstract | Crossref Full Text | Google Scholar