Automatic detection of facial expressions during the Cyberball paradigm in Borderline Personality Disorder: a pilot study

Borderline Personality Disorder (BPD) symptoms include inappropriate control of anger and severe emotional dysregulation after rejection in daily life. Nevertheless, when using the Cyberball paradigm, a tossing game to simulate social exclusion, the seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and contempt) have not been exhaustively tracked out. It was hypothesized that these patients would show anger, contempt, and disgust during the condition of exclusion versus the condition of inclusion. When facial emotions are automatically detected by Artificial Intelligence, “blending”, -or a mixture of at least two emotions- and “masking”, -or showing happiness while expressing negative emotions- may be most easily traced expecting higher percentages during exclusion rather than inclusion. Therefore, face videos of fourteen patients diagnosed with BPD (26 ± 6 years old), recorded while playing the tossing game, were analyzed by the FaceReader software. The comparison of conditions highlighted an interaction for anger: it increased during inclusion and decreased during exclusion. During exclusion, the masking of surprise; i.e., displaying happiness while feeling surprised, was significantly more expressed. Furthermore, disgust and contempt were inversely correlated with greater difficulties in emotion regulation and symptomatology, respectively. Therefore, the automatic detection of emotional expressions during both conditions could be useful in rendering diagnostic guidelines in clinical scenarios.

1 Introduction

Borderline Personality Disorder (BPD) is understood as a persistent pattern of difficulty with emotion regulation, impulse control, identity diffusion, interpersonal conflict, and social cognition impairment. These alterations in the recognition and differentiation of the self and others’ mental states, contribute to affective instability, particularly in social contexts (1). A common feature of BPD is the feeling of abandonment, even though it might not always be present. Patients tend to identify ambiguous pictures as angry faces (2), increasing the frequency at which they experience profound negative emotions, impulsive behaviors, and mistrust in their social interactions.

Nevertheless, the facial expression of patients with BPD during social interactions has been less often studied. Noteworthy, while Staebler et al. (3) evaluated the facial expression of emotions during the Cyberball paradigm, a virtual game to evaluate social interaction and ostracism (4), some issues remained uncovered in such study. In the present work, they were distinctly addressed.

In Staebler et al. (3), two coding systems were employed; one was based on judgment agreement and the second method was an interpretation of the Emotional Facial Action Coding System (EMFACS) under positive, negative, and mixed emotions by a computer program. Instead, a completely automatic analysis was used here. FaceReader (5) is the first commercially available Automated Facial Coding software to assess primary and secondary emotions in facial expressions as well as valence, arousal, head movement, and heart rate; it also renders high resolution and spares the manual coding and judgment agreement needed for the EMFACS implementation.

The term “condition of exclusion” in the Cyberball paradigm does not entail literal and absolute exclusion of the participant. While participants are indeed included in the game throughout the condition of inclusion, under the condition of exclusion, the first interval comprises inclusion, and then, the ostracism properly begins. In particular, Staebler et al. (3) did not evaluate the facial expressions of the complete Cyberball paradigm, as they evaluated them after the tenth ball throw; in consequence, the first period of the condition of exclusion, when the subject is still participating in the game, was not considered.

Williams (4) identified three stages of the time intervals of the Cyberball paradigm: reflexive, reflective, and resignation. The reflexive stage develops while the game is going on. Therefore, herein, analyses of emotional expression intensities throughout the game and its temporal segmentation during the reflexive stage were conducted in female patients with BPD. In Staebler et al. (3), a comparison was made among four different groups, two consisting of patients with BPD and two consisting of healthy controls. Hence, a repeated measure analysis of the non-verbal expression, including both conditions, was not carried out; thus, a complete temporal analysis of the Cyberball paradigm is lacking. Moreover, the joint effects of the conditions of inclusion and exclusion on facial expressions were in here explored using the FaceReader software.

Although patients with BPD were not compared with a control group, their patterns were randomly shuffled to create a synthetic control group for comparison, as in Chichilnisky (6); the methodology is detailed in Section 2.5.2. As stated by Staebler et al. (3), patients with BPD exhibited more blended facial expressions (displaying features of mixed and different emotions) and masking of emotions (covering a negative emotion with smiling) than healthy controls; likewise, even when masking was not statistically significant between conditions, this behavior was remarked during exclusion compared to inclusion. Here, the comparison of masked emotions between conditions was investigated.

In Staebler et al. (3), the percentage of instances of receiving the ball for the exclusion procedure was 13.3%, while in Gerra et al. (7) it was lessened up to 10%. Here, the protocol was designed to ostracize the subject with greater severity; therefore, the individual received the ball at a percentage of 6.6%.

Consequently, the following hypotheses were tested:

● Clinical measurements are correlated within themselves, i.e., symptomatology and dysregulation are positively correlated.

● Subjective ratings of the Need Threat Scale (NTS) are higher after the condition of exclusion versus inclusion.

● The intensity of negative emotions (anger, disgust, contempt, fear, and sadness) expressed during the condition of exclusion is higher compared to the condition of inclusion.

● The distributions of the patterns of emotions under inclusion and exclusion are different compared to a shuffled and randomized pattern.

● The second and third segments of the condition of exclusion versus the respective ones under the condition of inclusion trigger more negative emotions (anger, disgust, contempt, fear, and sadness).

● We expected blended and masked emotions to be different during the condition of exclusion versus the condition of inclusion.

● Some intensities of emotions, like anger and contempt, are positively correlated with the abovementioned scales, looking forward to finding useful markers of emotional dysregulation and symptomatology.

2 Method

2.1 Subjects

The Ethics Committee of the National Institute of Psychiatry “Ramón de la Fuente Muñiz”˜ (INPRFM) approved the project. Patients agreed to participate in the study by signing informed consent. The sample comprised 14 women with a confirmed diagnosis of BPD according to the DSM-IV diagnostic criteria, who regularly attended the Borderline Personality Disorder Clinic of the INPRFM. BPD patients were recruited in outpatient settings. BPD onset can appear in adolescence or early adulthood and the average age of the recruited patients was 26 ± 6 years old, ranging from 19 to 35 years old. They had an average schooling of 12.7 ± 4 years. Socio-demographic and clinical data is presented in Table 1.

Table 1

Table 1 Description of patients’ socio-demographic and clinical characteristics.

All of them were diagnosed with BPD, as stated above, and had received four sessions of a psychoeducational treatment. Only 3 out of the 14 women had received Dialectical Behavioral Therapy (DBT). Patients did not present any neurological disease or manic or psychotic episodes that would jeopardize their performance in this study. Patients were not agitated, aggressive, or suicidal during the study. None of the patients decided to quit without completing the evaluations. For all patients, except for one, Major Depressive Disorder (MDD) was the most frequent comorbidity of Axis 1; for the one excepted patient, Obsessive Compulsive Disorder was the respective comorbidity. In 9 out of the 14 patients, MDD was recurrent, and yet 13 out of the 14 subjects were in remission. One patient was relapsing, and in two of them, it was persistent.

Patients were prescribed a total of 33 medications. Of these, 48.48% (16/33) were antidepressants (Fluoxetine, Sertraline, Duloxetine, Venlafaxine, Fluvoxamine, Mirtazapine, and Citalopram), 24.24% (8/33) were antipsychotics (Quetiapine and Risperidone), 21.42% (7/33) were anticonvulsants (Magnesium Valproate, Topiramate, and Pregabalin), and 6.06% (2/33) were benzodiazepines (Clonazepam). All fourteen subjects took at least one antidepressant, 7 of them were also prescribed at least one antipsychotic, 5 were taking anticonvulsants, and 2 reported the consumption of at least one benzodiazepine.

2.2 Clinimetric evaluation

The subjects were evaluated individually using the following tests:

2.2.1 Structured Clinical Interview for DSM-IV Axis II Personality Questionnaire (SCID-II)

The SCID-II (8) is a semi-structured interview consisting of 119 yes/no questions. It can be used to formulate Axis II diagnoses, both categorical and dimensional. The psychometric properties of the SCID-II have been widely studied. According to a review of the Encyclopedia of Clinical Neuropsychology, Gorgens (9), SCID-II internal consistency coefficients ranged from 0.71 to 0.94 and inter-rater reliability coefficients ranged from 0.48 to 0.98 for categorical diagnosis. Also, inter-rater reliability on dimensional judgments ranged from 0.90 to 0.98. The validity of the SCID-II has been guided by the “LEAD standard”, based on longitudinal studies performed by experts. The diagnostic power of the SCID-II was 0.85 or greater for five personality disorders. Specifically, questions number 90 to 105 of the SCID-II assess symptoms for BPD (10).

2.2.2 Barratt Impulsiveness Scale (BIS-15)

BIS-15 is the currently widely used version of the Barratt Impulsivity Scale. It has been psychometrically validated in Spanish adults and adolescents (11). Internal consistency was 0.793 and test-retest reliability was 0.80. A three-factor structure was confirmed by factor analysis, accounting for 47.87% of the total variance in BIS-15 total scores. This is a self-report scale measuring three areas of impulsiveness: 1) attentional, 2) motor and 3) non-planning impulsiveness. This Spanish version consists in 15 items scored from 0 to 4.

2.2.3 Borderline Evaluation of Severity Over Time (BEST) (Version 1.7)

BEST is composed of 15 self-report items measuring the severity of the main symptoms of patients with BPD, e.g., mood reactivity, identity alteration, unstable relationships, paranoia, emptiness, suicidal thinking, and negative actions, on a five-point Likert scale (12). Cronbach’s α coefficients for patients with BPD and controls were respectively, 0.86 and 0.90. Test-retest reliability was moderate (r = 0.62, n = 130, p ≤ 0.001) (12).

2.2.4 Need Threat Scale

This scale consists of 14 items that evaluate the intensity of the person’s adverse experience during the game (13). The total score is 140, where higher score indicate more ostracism. The NTS was administered after each condition; i.e., inclusion and exclusion, as the most suitable method to measure subjective perception after the Cyberball paradigm administration. Convergent and discriminant validity studies showed subscales of the NTS and most of the ten Sheldon subscales (the Sheldon scale measures autonomy, relatedness, competence, self-esteem, popularity-influence, physical thriving, self-actualization-meaning, money-luxury security and pleasure-stimulation) were correlated (α = 0.71 to 0.79); factor analyses found the four-factor structure of the NTS was not supported and thus the four needs seemed to be overlapped rather than distinct (14).

2.2.5 Difficulties in Emotion Regulation Scale, Spanish version, DERS-E

This scale comprises 24 items that evaluate the capacity of being dysregulated in four domains:

Nonacceptance of Emotional Responses, Difficulties Engaging in Goal-Directed, Lack of Emotional Awareness, and Lack of Emotional Clarity. The subscales of Cronbach´s α range from 0.85 to 0.68 according to Hervás and Jódar (15). The validity through contrasted groups and the correlation with concurrent scales showed significant results (Pearson’s r coefficient ranging from 0.51 to 0.76, p< 0.05).

2.3 Materials

2.3.1 The Cyberball paradigm

Version 5 of the Cyberball sotware was used in the study, Previous versions of the software can ran of PC and/or Mac platforms while this version provides an online executable alternative. The participants were scheduled to be online, playing with two virtual players. The data from the game, including the total number of players’ ball throws, and the number of mouse clicks, were collected for analysis by the Cyberball software itself (the information on the number of clicks was not considered for this study). The game consisted of throwing a ball with two additional players. However, the participants were interacting with programmed virtual players, despite their initial perception of playing with real people.

In the present study, each condition of the Cyberball paradigm was divided into three temporal segments. Such segmentation was implemented to discriminate and consider the moments when participants were effectively excluded, specifically under the condition of exclusion. The average duration of the Cyberball paradigm for the current participants was 2 minutes and 25 seconds. However, the initial 14 seconds were disregarded given that time was used by the experiment monitor to type the names of the participants and select the respective experimental condition in the game. Therefore, the recordings displayed two minutes and 9 seconds of relevant material which was segmented as follows:

● segment 1: from 0:00 to 0:59 minutes, during which the patient’s facial expressions under the respective experimental condition are displayed, 45 seconds; i.e., from 0:14 to 0:59 minutes;

● segment 2: from 1:00 - 1:45 minutes, during which the patient’s facial expressions under the respective experimental condition are displayed, 45 seconds;

● segment 3: from 1:46 - 2:25 minutes, during which the patient’s facial expressions under the respective experimental condition are displayed, 39 seconds.

During the condition of inclusion, a total of 30 ball throws were made. One third of the ball throws were programmed to be received by the patient. In the first, second, and third segments, the ball was thrown to be caught by the participant a total of four, four, and two times, respectively. Regarding the condition of exclusion, the number of 30 ball throws remained unchanged; notwithstanding, only two of them were programmed to be received by the participant during this part of the experiment. The occasions in which the participant received and threw the ball occurred in the first segment exclusively.

Each participant could decide whom to throw the ball to among the other two players and when to do it. The NTS was conducted after the culmination of each experimental condition. As previously mentioned, in the Cyberball paradigm, the participant must throw the ball toward one of the two other players, this is emphasized since leads to time variations in the final duration of the game and, consequently, affects the duration of the video recordings, which ranges from 2 minutes and 5 seconds to 2 minutes and 32 seconds.

2.3.2 FaceReader 7.0

The software first detects a person’s face based on the Viola and Jones algorithm. Subsequently, two combined methods are applied. The first method, Face modeling and classification, is based on an Active Appearance Model (AAM) which, detects over 500 key points of the face and refers to previous databases to estimate the variations of the new face image sampled compared to an average face (5). The second method, Deep face classification, uses a deep artificial intelligence algorithm of pattern recognition to identify facial expressions according to (5). Afterwards, the Principal Component Analysis (PCA) is applied to reduce the model dimensions. With the gathered information, a neural network is trained to classify the emotions.

To train the neural network, 10000 manually rated images were considered. The Deep face classification method uses deep face classification from image pixels and can be used alternatively when the Face modeling and classification method provides no informative results. These models calculated the probability and intensity scores for facial expressions on a continuous scale from 0 to 1 (5). FaceReader accurately identifies facial expressions, with a reported agreement between manual and automatic detection ranging from 89.6% for scared expressions to 99% for happy expressions (5).

In the absence of a neutral face captured prior to the experimental conditions, a continuous calibration was performed. This procedure allows the adaptation to the particular face bias. According to the FaceReader’s manual (5), this software continuously averages the facial expression intensities, correcting them. Therefore, the intensity of the current frame considers the average intensity of previous frames as follows (Equation 1):

$\begin{array}{ll}\text{Expression\hspace{0.17em}intensity\hspace{0.17em}}=\text{\hspace{0.17em}max}\left(0,\frac{l_a-l_m}{1-l_a}\right),& (1)\end{array}$

where $l_a$ is the expression intensity in the current frame and $l_m$ is the average expression intensity over all frames before the current frame. If $\frac{l_a-l_m}{1-l_a}<0$, the previous equation is equal to 0; i.e., the expression intensity is also zero.

Additionally, the intensity of neutral is given by (Equation 2):

$\begin{array}{ll}\text{Intensity\hspace{0.17em}Neutral\hspace{0.17em}}=\frac{N_a+(1-l_{\max m})}{2},& (2)\end{array}$

where $N_a$ is the intensity of Neutral classified by FaceReader in the current frame and $l_{\max m}$ denotes the maximum average intensity of all emotions in all the frames before the current one.

The FaceReader software classifies facial expressions into seven categories: happiness, sadness, anger, surprise, fear, disgust, and contempt; i.e., the primary emotions.

2.4 Procedure

The patients completed the clinimetric evaluations several days before the study began. If they met the inclusion criteria, they were invited to participate in this investigation. Under both experimental conditions, the patients with BPD were carefully video recorded with an SJCAM HD 1080P camera, which was located in a special room at the INPRFM illuminated with a white light bulb.

Standard instructions were given about the need to visualize the scenario in which the other players would be present, etc. (13). Subjects were seated 50 cm away from a 14-inch screen. When participants had the chance to throw the ball, they could choose either player 1 (on the left) or player 3 (on the right) by clicking on them with the mouse. After instructing the patients, the experiment monitor started to record the video and went out of the room. Once the patient was in private, the game began and the recording was stopped until the end of the game. The general experimental procedure is illustrated in Figure 1.

Figure 1

Figure 1 Experimental protocol. Participants were video recorded while playing the Cyberball paradigm under each condition of the game, i.e., inclusion and exclusion. Experimental conditions were video recorded separately. The videos were subjected to the FaceReader algorithms. The values of primary emotions of one patient with Borderline Personality Disorder are displayed in the two graphics, corresponding to (A) inclusion and (B) exclusion, presenting the temporal segmentation. Two main analyses (not shown) were conducted: 1a) second-by-second pattern analyses (percentage of subjects displaying the highest intensity of a particular emotion in that second) for each condition; 1b) pattern analysis of the sum of the percentages of 15 seconds and, the difference of these percentages between conditions (exclusion minus inclusion). 2) Adjusted area under the curve: 2a) individual and 2b) group analyses.

2.5 Statistical analysis

2.5.1 Socio-demographic data and clinimetric assessments

Data from the scales and the collected patterns were analyzed using Excel and the Statistical Package for the Social Sciences (SPSS) v.17. Frequency and percentages were calculated for the categorical variables, while means and standard deviations were computed for the scales. Spearman’s correlations were conducted between scores of scales (BEST and DERS-E, BEST and BIS-15 and BIS-15 and DERS-E). Afterwards, these scales were correlated with the AUC of each emotion. The reported values from the variables recorded by the FaceReader software were analyzed using the following two methods:

2.5.2 Pattern analysis: second-by-second pattern and temporal segmentation analyses

Given that emotions can be considered as emotional states if they last at least 0.5 seconds (16), a second-by-second descriptive analysis was carried out with the original outputs of the signals, as shown below. The software can be set up to store and process 1, 15, 30, or 60 frames per second. For this study, the FaceReader software was set to default mode, thus the frame rate was automatically determined by the program configurations (16). Therefore, 30 frames per second were recorded for 12 of the subjects, while 60 frames per second were captured for the remaining two patients. Considering the complete videos recorded from the 14 participants’ faces, the total number of frames per second ranged from 5,362 to 12,878 for both conditions.

For each emotion, we averaged the 30 or 60 scores reported, depending on the frame rate used, to obtain a single value representing the emotion score per second. After calculating the emotion score per second of each emotion, the patient’s most intense emotion for each second was defined as the emotion with the highest score in the respective second. The above procedure was applied to the outputs of the video recordings of the faces of all 14 participants under each experimental condition. Afterwards, for each condition, the percentage of patients displaying each primary emotion was determined second by second. To streamline the analysis, we averaged the percentage of patients’ emotions every 15 seconds (except the last interval, which consisted only of 10 seconds), dividing the first segment into four intervals, while the second and third segments were divided into three intervals each.

In each interval, the most and least common emotion was found. Finally, it was determined which emotion showed the greatest increase and the greatest decrease in each patient due to the social exclusion for each interval. Therefore, for each emotion considered, the difference between the percentage of patients who showed the respective emotion during exclusion and the one during inclusion was obtained.

As previously stated, we aim to investigate whether the distributions of the patterns of emotions under inclusion and exclusion are different compared to a shuffled and randomized pattern. Consequently, the reported frequencies under each experimental condition were shuffled second by second, obtaining a white noise version of them to perform the comparisons.

Shuffling analyses create new patterns by randomizing the data sets. The patterns of both conditions were tested against their respective random-shuffled data sets created by randomly shuffling the frequencies of the original data sets, as shown in Algorithm 1. White noise is inherently random; furthermore, each white noise sample is statistically independent of the others, and there is no correlation between successive samples. This randomness property makes white noise useful in various applications, such as modeling stochastic processes, simulating random events, and providing a baseline for comparison in statistical analyses (6, 17–20).

When used in data analysis or modeling, white noise is sometimes assumed as a null hypothesis; i.e., a synthetic control group where the patterns observed in the data are due to random variation inherent in the data per se, rather than any underlying factors, such as a dynamical system attractor. Deviations from white noise behavior in a signal might indicate the presence of underlying patterns, trends, or systematic effects that warrant further investigation. Additionally, techniques such as spectral analysis can be employed to examine the frequency content of a signal and assess its deviation from the characteristics of white noise (18–20)).

Algorithm 1. Shuffled matrix for the comparison of the patterns of both conditions.

2.5.3 Area under the curve (AUC) analysis

A subject-per-subject standardized analysis with the seven emotions was carried out, and the AUC was obtained. To determine the degree of masking, the AUC respective to happiness was summed to the AUC of each emotion, independently. Group differences were sought between and within conditions and time intervals for every emotion. For the implementation of the AUC analysis, the reported FaceReader outputs were processed according to the following procedure:

1. Plotting facial expression profile recorded during the conditions of inclusion and exclusion.

2. Determining the total number of frames from the video recorded during each experimental condition, exclusively while the game was running; i.e., disregarding the video’s initial part when patients typed their names and chose the respective experimental condition. As previously mentioned, each video had a variable duration. Therefore, the total number of frames from each video is the product of the video length in seconds and the frame rate at which the video was recorded, 30 frames per second (n = 12 participants) or 60 frames per second (n = 2 participants).

3. Calculating the Area Under the Curve (AUC) according to the trapezium method (21) for each emotion and during the respective experimental conditions.

4. Standardizing the time interval used to compute the AUCs by multiplying the emotion intensity score per second and the number of frames.

The above protocol was implemented to every participant’s data. Additionally, differences between the emotions expressed during both experimental conditions, with no segmentation, were sought by performing a one-way repeated measures ANOVA and Wilcoxon tests. The AUC for every emotion expressed during each of the three segments was computed, and a two-way repeated-measures ANOVA was performed to assess differences between conditions per segment. Afterwards, pairwise comparisons (paired Student’s t-test) with Bonferroni corrections were conducted, to explore differences between segments.

Data normality was verified by conducting Shapiro-Wilk tests and Q-Q plots, as implemented in rstatix and ggpubr R packages. In cases the normality assumption was not met, data normalization was performed using the bestNormalize R package to determine the optimal transformation for each data set. Normalizations were carried out according to hyperbolic arcsine transformation for anger, happiness, sadness, and surprise; ordered quantile normalizing transformation for fear; and Yeo-Johnson transformation for disgust. Outliers were identified using the identify_outliers function from the rstatix R package. Outliers are defined as data points that fall above the Q_{3 +} 3IQR or below the Q₁ − 3IQR thresholds, where Q₁ and Q₃ denote the first and third quartile and, IQR represents the interquartile range. For these analyses, ANOVAs were performed with and without extreme outliers, with and without the Benjamini-Hochberg corrections, respectively. In general, both analyses yielded the same results with and without outliers. Therefore, the herein-reported results include outliers.

All the analyses presented in this section were performed using R (version 4.1.0 – “Camp Pontanezen”) R Core Team (22). Given that multiple ANOVAs were performed, the p-values were corrected using the Benjamini-Hochberg method. This correction was implemented using the p.adjust function, from the stats R package, to control the false discovery rate (FDR).

3 Results

3.1 Correlations between scales

In Table 2, Spearman’s correlations between clinimetric scales are reported. The SCID-II scores are significantly correlated with the BIS-15 (r = 0.37, p< 0.05), BEST (r = 0.65, p< 0.01), and DERS-E (r = 0.68, p< 0.01) scales. Additionally, the BIS-15 scale is significantly correlated with the DERS-E (r = 0.23, p< 0.05) scale. Similarly, the scales BEST and DERS-E were positively correlated (r = 0.32, p< 0.05).

Table 2

Table 2 Spearman’s correlations between the clinical scales, with the Bonferroni correction for the p-value.

3.2 Need threat scale

The mean and standard deviation scored on the NTS for the condition of inclusion were 78.50 ± 29, while the corresponding scores on the NTS for the condition of exclusion were 110.57 ± 29. The statistical difference between them was significant (t₁₃ = 5.25, p< 0.0001).

3.3 Cyberball

There were no significant differences between the duration, in seconds, of both experimental conditions (condition of inclusion = 129.92 ± 11; condition of exclusion = 128.40 ± 2, t₃ = 0.12, p< 0.90).

3.4 Second-by-second analysis: frequencies of emotions according to their highest mean intensity

In Table 3, the frequencies converted to percentages of the pattern analysis are shown. Regarding the condition of inclusion, in the first segment (seconds 1 to 59): sadness was the most common emotion displayed during the first (seconds 1 to 15), second (seconds 16 to 30), and fourth (seconds 45 to 59) intervals, whereas contempt was the most common during the third (seconds 31 to 44) interval. Concerning the condition of exclusion, in the first segment, sadness was also the most common emotion only during the first (seconds 1 to 15) and fourth interval (seconds 45 to 59), in contrast to the condition of inclusion. Likewise, contempt was also the most common during the second (seconds 16 to 30) and third intervals (seconds 31 to 44). During the second segment (seconds 60 to 106) of the condition of inclusion, sadness was the most predominant emotion while fear was the least expressed emotion during the entire segment. Respecting the second segment (seconds 60 to 106) of the condition of exclusion, sadness was the most common emotion displayed during the second (seconds 76 to 90) and third (seconds 91 to 106) intervals; however, during the first interval (seconds 60 to 75), contempt was the most common emotion. Regarding the third segment (seconds 106 to 145) of the condition of inclusion, sadness was reported as the most common emotion displayed during the first (seconds 106 to 120) and third (seconds 136 to 145) intervals; similarly, during the second interval (seconds 121 to 135), anger was reported as the most expressed emotion. In the condition of exclusion, third segment (seconds 106 to 145), sadness was the most common emotion during the second (seconds 121 to 135) and third (seconds 136 to 145) intervals, while happiness was the most common during the first interval (seconds 106 to 120). The increases in fear, surprise, and happiness, as well as the decreases in sadness and anger at the end of the condition of exclusion, contrasted to that of inclusion one, were evident. Additionally, sadness held the highest value throughout the entire game.

Table 3

Table 3 Summary of the pattern of emotions.

In Table 4, the differences in percentages of the reported values under the condition of exclusion minus the ones under the condition of inclusion, per emotion, are presented for each segment and interval. Differences, in absolute value, were greater than ten percentage points; i.e., all differences are greater than 10 or less than -10. In the first segment, contempt attained the greatest increase, whereas surprise and happiness held the greatest increase during the last two intervals. Conversely, anger and contempt reported the greatest decreases, with anger showing greater consistency than contempt. In Figure 2, the percentages for the group, per emotion, during the entire game can be visualized.

Figure 2

Figure 2 Pattern of emotions in the Cyberball paradigm during (A) the condition of inclusion and (B) during the condition of exclusion across time. Happiness (yellow), sadness (blue), anger (green), surprise (purple), fear (black), disgust (orange) and contempt (gray). Each bar represents 1 second. For each emotion, it shows the percentage of patients with Borderline Personality Disorder who displayed such emotion as the one scoring the highest value in that second. Sadness (blue) was constantly expressed. Anger (green) increased throughout the condition of inclusion and it decreased in the condition of exclusion. The prominence of happiness (yellow) and surprise (purple) suggested a higher degree of patients showing those emotions during ostracism.

Table 4

Table 4 Pattern analysis.

This can be more easily visualized in Figure 3, where the differences between the frequencies of the condition of inclusion minus the ones of exclusion were calculated.

Figure 3

Figure 3 The second-to-second differences between the pattern analyses of the inclusion minus exclusion conditions among participants with Borderline Personality Disorder during the Cyberball paradigm were examined, with frequencies transformed to percentages. An increase in the percentage of participants expressing anger (green) was particularly notable, especially toward the end of the inclusion condition (top). Conversely, in the exclusion condition (bottom), surprise (purple) and happiness (yellow) were more predominantly expressed.

The comparative error was remarked due to sample sizes. The small sample sizes were crucial, and even the greatest change of the third interval of happiness, which raised from 8.40% in inclusion to 32.70% in exclusion, lacked statistical significance (comparative error = 28.54, percentage difference = 24.30). Under the assumption of 20 hypothetical subjects, the observed difference became statistically significant (comparative error = 23.88, percentage difference = 24.30).

In Figure 4, the distributions of the shuffled frequencies are represented. Almost all the distributions per emotion and condition and their randomized patterns were statistically significant.

Figure 4

Figure 4 (A). Inclusion Patterns of the primary emotions 1) Happiness, 2) Sadness, 3) Anger, 4) Surprise, 5) Fear, 6) Disgust, and 7) Contempt. (B). Shuffled Inclusion shows that the patterns in A are lost. 1)Happiness vs. Shuffled Happiness (t = 2.41, p< 0.01); 2) Sadness vs. Shuffled Sadness (t = 10.55, p< 2.27611E−20); 3) Anger vs. Shuffled Anger (t = 5.54, p< 1.07248E−07); 4) Surprise vs. Shuffled Surprise (t = 4.41, p< 1.78844E − 05); 5) Fear vs. Shuffled Fear (t = 9.66, p< 6.45638E − 18); 6) Disgust vs. Shuffled Disgust (t = 9.89, p< 1.55306E − 18); and 7) Contempt vs. Shuffled Contempt (t = 5.79, p< 3.16769E − 08). More details in the text. (C). same as A for exclusion patterns (D). Shuffled Exclusion shows that the patterns in A are lost. 1)Happiness vs. Shuffled Happiness (t = 3.76, p< 0.0002), 2) Sadness vs. Shuffled Sadness (t = 9.34, p< 1.04266E − 16), 3) Anger vs. Shuffled Anger (t = 1.5912, p< 0.11), 4) Surprise vs. Shuffled Surprise (t = 1.07, p< 0.28), 5) Fear vs. Shuffled Fear (t = 9.38, p< 8.28122E − 17), 6) Disgust vs. Shuffled Disgust (t = 10.91, p< 7.29614E − 21), and 7) Contempt vs. Shuffled Contempt (t = 4.06, p< 7.67073E − 05). More details in the text.

3.5 Area under the curve

3.5.1 Individual level

As shown in Figure 5, different proportions of emotions were found for each of the three segments in every condition, noting several cases of blending and masking. Regarding participants 5D and 5L, anger was predominantly present, while contempt was found in high percentages among several subjects (5A, 5B, 5C, 5F, 5G, and 5I), in the condition of inclusion. Sadness was expressed in many subjects in both conditions (5G, 5H, 5K, 5M, 5N). Some participants (5B, 5F,5L, 5M, 5N) expressed high levels of happiness in both conditions, inclusion and exclusion.

Figure 5

Figure 5 Pie charts displaying the proportions of the area under the curve (AUC) for the seven emotions. The AUC were obtained after the facial emotions of patients with Borderline Personality Disorder were detected by the FaceReader software during the Cyberball paradigm session. Each participant is represented by a capital letter of the alphabet (from letter A–N). The proportions of the AUC were obtained for the three time segments and the two conditions. The conditions of inclusion and exclusion were analyzed separately in three AUCs: 1 minute for the first segment (left), and 45 seconds each for the second and third segments (middle and right, respectively). The expression of facial emotions of patients are depicted for social inclusion (above) and social exclusion (below). Each color represents one emotion: yellow is happiness; blue is sadness; anger is green; surprise is purple; fear is black; disgust is orange; and gray is contempt.

3.5.2 AUC: group level for both conditions

In Table 5, the values of the AUC for each emotion and masked emotions are shown. Wilcoxon non-parametric tests were conducted to compare the condition of inclusion versus the one of exclusion. Happiness, anger, and surprise, as can be seen in Figure 6, were the emotions with the greatest changes, while sadness, fear, disgust, and contempt remained nearly constant across both experimental conditions; fear and disgust were the emotions with the lowest mean values reported. Happiness and surprise exhibited higher levels during exclusion; on the contrary, anger followed the opposite pattern, attaining higher levels during inclusion than during exclusion. However, no statistically significant differences were found between conditions, primarily due to large variations across subjects. Only surprise exhibited a tendency toward significance (Z = -1.72, p< 0.08).

Figure 6

Figure 6 Mean values and standard errors of the total areas under the curve for each emotion during the condition of social inclusion (black lines) and social exclusion (red lines) while patients with Borderline Personality Disorder performed the Cyberball paradigm. Although surprise was nearly significant (p< 0.08) there were no significant differences between emotions across conditions. Significant differences between emotions of the same condition were represented with brackets for the condition of inclusion (A) and for the condition of exclusion (B) (**, p< 0.001). In both conditions, sadness, happiness, and contempt were the emotions with the highest values, and fear and disgust were the emotions with the lowest values. Overall, during exclusion, statistically greater significant differences between emotions were found.

Table 5

Table 5 Means and standard errors (SE) of the calculated Areas Under the Curve (AUC) of each facial emotion expression for the conditions of social inclusion and exclusion and their comparisons and similar measures now for masked emotions are presented.

Therefore, considering the polarized levels of each emotion, comparisons within the conditions were conducted between emotions. For inclusion, as observed in Figure 6, sadness, anger, and contempt were the emotions reporting the higher values, with mean AUC scores of 300.23 ± 306, 228.74 ± 400 and 204.70 ± 176, respectively. Happiness, surprise, fear, and disgust: 186 ± 255, 74.20 ± 120, 44.85 ± 46 and 35.51 ± 33, respectively, were the emotions that exhibited middle to lower mean values. Respecting the condition of inclusion, among the 21 possible combinations of emotions, significant differences were observed in the following pairs: happiness and fear (Z = -2.22, p< 0.02); happiness and disgust (Z = -2.22, p< 0.02); sadness and surprise (Z = -2.54, p< 0.01); anger and fear (Z = 2.41, p< 0.01); contempt and surprise (Z = -1.97, p< 0.04), sadness and fear (Z = -3.29, p< 0.001); sadness and disgust (Z = -3.29, p< 0.001); anger and disgust (Z = -3.29, p< 0.001); fear and contempt (Z = -2.73, p< 0.006); contempt and disgust (Z = -2.85, p< 0.004).

The emotions with significant differences for the condition of exclusion were as follows: fear and anger (Z = -2.04, p< 0.04); contempt and anger (Z = -2.29, p< 0.02), happiness and fear (Z = -2.66, p< 0.008); happiness and disgust (Z = -2.79, p< 0.005); sadness and anger (Z = -3.17, p< 0.002); sadness and fear (Z = -3.29, p< 0.001); sadness and disgust (Z = -3.23, p< 0.001); anger and disgust (Z = -3.23, p< 0.001); fear and contempt (Z = -3.23, p< 0.001); and contempt and disgust (Z = -3.23, p< 0.001).

In summary, respecting the condition of inclusion, five out of ten significant differences exhibited a p< 0.001, while in the condition of exclusion, eight out of the ten significant differences exhibited a p< 0.001. Therefore, considering the emotions with no segmentation, a noticeable contrast between emotions was found during the condition of exclusion compared to the condition of inclusion. During the condition of exclusion, the higher means of the AUC were sadness and happiness with 320.20 ± 353 and 257.26 ± 310, respectively, while contempt, surprise, anger, fear, and disgust, with 193.72 ± 148, 128.46 ± 128, 89.58 ± 39, 50.88 ± 37 and 35.37 ± 38, respectively, were the four emotions with the lower values. The black square brackets in Figure 6A represent the significant differences in emotions during inclusion. Analogously for the condition of exclusion, in Figure 6B, the square brackets below represent the significant differences.

3.5.3 Summed values of the AUC: masking of emotions

The summed AUC of happiness and surprise yielded the only statistically significant result (Z = −2.41, p< 0.01), indicating an increase for exclusion versus inclusion. The mean AUC was 260.84 ± 227 for inclusion and 385.73 ± 285 for exclusion (Table 5). The individual values ranged from 6.72 to 1003.53 during the condition of inclusion and from 7.82 to 1086.46 during the condition of exclusion.

3.5.4 Differences between the AUC of segments at group level

The analyses revealed surprise was statistically significant between conditions, and fear was near-significance, see Table 6. Notably, anger showed a significant interaction: it increased during inclusion and decreased for exclusion across segments (Figure 7). However, the power of each test, considering outliers, was low due to the small sample size: anger (0.18), disgust (0.05), happiness (0.10), sadness (0.05), fear (0.28), and surprise (0.08).

Figure 7

Figure 7 The Area Under the Curve (AUC) of the primary emotions (A) Anger, (B) Happiness, (C) Contempt, (D) Disgust, (E) Sadness, (F) Fear, and (G) Surprise in Borderline Personality Disorder expressions during the conditions of inclusion or exclusion in a Cyberball paradigm session, divided into time segments (1 to 3), were analyzed using box plots with outliers represented by black dots. Two-way repeated measures ANOVAs were performed to assess differences between conditions per segment. Anger demonstrated a significant interaction between the third segments of the conditions, being higher during inclusion than during exclusion (p< 0.01).

Table 6

Table 6 Analysis of the area under the curve of each emotion by segments: Details of the two-way Repeated-Measures ANOVA for each analyzed variable, including outliers, are presented.

3.5.5 Correlations between AUC

There were few significant correlation between AUC. Specifically, the AUC of happiness during exclusion was negatively correlated with the AUC of anger also during that condition (r = −0.56, p< 0.03). Similarly, the AUC of happiness was correlated between conditions (r = 0.65, p< 0.01); happiness during inclusion was positively correlated with disgust during exclusion (r = 0.55, p< 0.04). Fear during inclusion was positively correlated with happiness during exclusion (r = 0.58, p< 0.02). The highest positive correlations, however, were observed between fear across conditions (r = 0.70, p< 0.005) and contempt also during both conditions (r = 0.72, p< 0.003). Finally, sadness during inclusion and contempt during exclusion were negatively correlated (r = −0.53, p< 0.04).

3.5.6 Correlations between AUC and clinimetric scales

In Table 7, significant correlations are outlined. There was a significant and positive correlation between the SCID-II and surprise during inclusion (r = 0.54, p< 0.05). In contrast, a significant negative correlation was observed between SCID-II and contempt during exclusion (r = −0.53, p< 0.05). See Figure 8. Additionally, consistent with the aforementioned result, the BIS-15 and the AUC of disgust during both conditions were negatively correlated, with statistical significance reported only during the condition of inclusion (r = −0.57, p< 0.03). Moreover, as demonstrated in the respective table, specifically, the AUC of disgust during the condition of exclusion and DERS-E exhibited a significant negative correlation (r = −0.56, p< 0.05). Furthermore, the NTS was almost positively correlated with sadness during exclusion.

Table 7

Table 7 Spearman’s correlations with Bonferroni corrections between the scores of the Structured Clinical Interview for DSM-IV (SCID-II), Barratt Impulsiveness Scale (BIS-15), Borderline Evaluation of Severity Over Time (BEST), the Need Threat Scale (NTS), and the values of areas under the curve per emotion are presented.

Figure 8

Figure 8 The scores of the Borderline Evolution of Severity Over Time (BEST) were positively correlated with the values of the Area under the curve (AUC) of surprise during the condition of inclusion (A). In contrast, scores of the Barratt Impulsiveness Scale (BIS-15) were inversely correlated with the values of the AUC of disgust during inclusion (B). Scores of the Structured Clinical Interview for DSM-IV (SCID II- BPD) were inversely correlated with the values of the AUC of contempt during exclusion (C). Additionally, scores of the Difficulties in Emotion Regulation Scale, Spanish version, (DERS-E) were inversely correlated with the values of the AUC of disgust in the condition of exclusion (D). Finally, scores of the Need Threat Scale (NTS) were positively correlated with the values of the AUC of sadness in exclusion (E).

4 Discussion

The present research provided a detailed follow-up of single and mixed facial emotional expressions under two levels of analysis. This is the first study to present the automatized analysis of facial expression of patients with BPD during the experimental conditions of the Cyberball paradigm.

The only report on a longitudinal evaluation of emotions has been conducted by Williams (4), who utilized a “feelings dial” to facilitate the measurement of emotions on a Likert scale ranging from happy to sad. This study employed a second-by-second analysis of decision-making processes and found that, in individuals without psychiatric disorders, mood began to decline after 20 seconds without the ball.

To our knowledge, no other study has attempted to describe in detail the distribution of emotions and the configuration of blended or masked emotions involved in the reflexive stage, as mentioned by Williams (4), not only after the tenth ball throw (3) but throughout the entire game.

Previous studies have highlighted an apparent contradiction as one of the features of the disorder. In the present study, anger exhibited a significant interaction with each condition and segment, increasing during inclusion while decreasing during exclusion. Therefore, this specific emotion, documented as challenging to manage according to the literature on people with BPD (23), was observed during the condition of inclusion. The interaction of anger occurred during the third segment of each condition, even though patients felt more ostracized in the condition of exclusion compared to the condition of inclusion (NTS).

Social exclusion induced by the Cyberball paradigm has been consistently shown to elicit intense negative emotional reactions, such as ostracism, in individuals without psychiatric disorders (4, 24, 25). According to self-reports, patients with BPD are particularly susceptible to exclusion (3, 7, 26). However, during the Cyberball paradigm, which might shed light on the specific processing of emotional stimuli in BPD, they tend to underestimate their participation percentage in the game when they are included, as indicated by the NTS (3, 7). Recent research on the recognition of faces, considering effortful control, found that students with high BPD features and low control tended to accurately detect and identify subtle negative emotions. This supports the “empathy paradox” hypothesis. However, they failed to detect most neutral faces, labeling them as negative, which favors the hypothesis of a negativity bias (27).

This subjective bias among patients with BPD has also been corroborated by physiological responses observed during the Cyberball game. Lower levels of Respiratory Sinus Arrhythmia (RSA) were documented during the inclusion condition compared to an eyes-open resting state, while RSA levels remained consistently low during the exclusion condition (7). In such study, patients exhibited lower baseline RSA levels than both controls and patients with MDD prior to the start of both conditions. These findings were interpreted in light of the challenges individuals with BPD encounter when engaging in social interactions, stemming from an initial heightened activation response. Consistent with the hypothesis of the emotional modulation paradox in patients with BPD, reports of autonomic nervous system activations associated with fight-or-flight responses (i.e., increased heart rate) have been observed during both the Cyberball paradigm and interviews, rather than a predominant involvement of the ventral vagal parasympathetic branch, which is linked to the social engagement system (28).

The AUC of surprise when the segmentation was carried out and the masked emotions of surprise plus happiness increased during social exclusion. During this condition, individuals with BPD may appear to anticipate ostracism. Therefore, the expectation of experiencing greater surprise during this condition might seem counterintuitive. Nonetheless, the emotion of surprise was significant between conditions and also when it was masked. These reactions could be explained by some of the assumptions made by (7). Patients with BPD perceive inclusion as a challenging and threatening situation. As a result, the “vagal brake” is withdrawn, and anger is triggered, particularly during the third segment of inclusion.

Fight, flight, and freeze responses may be present, while social responses are inhibited. As the exclusion condition begins, patients’ emotions apparently remain unchanged until the participant suddenly stops receiving the ball. Surprise and happiness may serve as markers of the intention for social engagement, presumably accompanied by increases in RSA. Conversely, anger nearly dissipates.

Traditional literature indicates that the effects of ostracism in individuals without psychiatric disorders are akin to pain (4). During the Cyberball paradigm, control subjects exhibited higher levels of happiness (both felt and unfelt) compared to patients with BPD (3), although the differences between conditions were not significant.

It is noteworthy that Staebler et al. (3) found a higher prevalence of blended emotions in patients with BPD compared to healthy controls, particularly during the condition of exclusion (BPD group: 74% vs. control group: 18%). It is important to note that Staebler et al. (3) evaluated emotions both separately and jointly, allowing for the identification of unique emotions as well as mixed emotions (defined as “two emotional expressions displayed at the same time”). In contrast, in our study, emotions were detected and reported continuously throughout the game. In this sense, the different methodologies, the single but longitudinally evaluated group and the higher level of ostracism observed in the current study, compared to that of Staebler et al. (3), may be responsible for the discrepancies of the different percentages of blended emotions between conditions.

In the present study, during the condition of exclusion, anger was found to be negatively correlated with happiness.

Contempt and disgust were present in both conditions in a similar intensity, exhibiting disgust a negative correlation with impulsivity during inclusion and likewise with BPD symptomatology and difficulties in emotion regulation, during the condition of exclusion. Culicetto et al. (29) propose disgust as a transdiagnostic index of mental illness across various pathologies. According to their review, in BPD, poor recognition of others’ disgust is associated with increased activation in the insula and posterior cingulate cortex.

Schienle et al. (30) found that women diagnosed with BPD perceive facial expressions of disgust in a higher rate than their healthy counterparts, but solely in instances where the individual exhibiting this emotion was male. Additionally, the spectrum of self-disgust, trait disgust, and disgust recognition were positively associated with disorder severity. However, although the present results seem to contradict these findings, it must be recalled that in the present study the inverse correlation between expressed disgust was found under the condition of inclusion and previously measured impulsivity; furthermore, the negative associations between disgust and dysregulation and contempt and severity happened especially when social exclusion was being undertaken. Anger had disappeared and the patients seemed to be overwhelmed by feelings of sadness. These results seem to signify severe symptomatology is associated with a decrease in the expression of disgust during inclusion. Likewise, this relationship remains unchanged during ostracism, and the patient with BPD behaves in a similar way when expressing contempt.

In patients with BPD, Kot et al. (31) found that self-disgust is highly associated with alexithymia, emotion regulation, and comorbid psychopathology but with a lower degree of disgust sensitivity. In these subjects, Unoka et al. (32) found a higher attribution of disgust and surprise than a control group when the task consisted in recognizing Ekman faces.

Given that most of the patients in the present research underwent psychoeducational therapy, and a few received various modules including tolerance distress and emotional regulation, i.e., DBT, a greater expression of emotions than patients without treatment could be expected, which might be related to lower disorder severity. In this sense, if emotional dysregulation consists in alterations in the identification of emotions, the nonacceptance of emotional responses, in difficulties of engaging in goal-directed, in the lack of emotional awareness, and lack of emotional clarity, then, especially disgust and contempt may be attenuated when all these features are present in a greater proportion.

The study by (33) stated that patients with BPD were more prone to rate disgust in questionnaires when viewing images of the International Affective System than control subjects. A lower activation of the left amygdala and an increased activation of the dorsolateral prefrontal cortex and the ventral striatum were found. It could be assumed that aberrant processing of emotions among frontal and limbic regions may underlie not only emotional processing when rating images but also its expression during a relatively socially complex task such as the Cyberball paradigm.

Moreover, our findings suggest the presence of additional interactions between the scales and emotions of patients with BPD. For instance, the correlation between surprise during the condition of inclusion and the score of the BEST may indicate that greater surprise when being equally included is associated with increased severity. As stated before, the inverse correlation between disgust during the condition of inclusion and the score of the BIS-15 might indicate that the expression of disgust is attenuated when impulsivity is presented, but in the case of surprise, there is a straightforward association.

In previous studies, the observation of happiness in healthy controls seems to be a positive indicator rather than a negative one. In preliminary studies (34, 35), a higher level of happiness was found in both conditions in patients with BPD with medium indexes of mistrust. Future studies should explore whether individuals with BPD who display higher levels of happiness are more likely to experience faster improvements in therapy compared to those with explicitly higher levels of social mistrust.

Additionally, Staebler et al. (3) observed more blending of emotions in exclusion than in inclusion across the four groups. However, Staebler et al. (3) did not report the precise percentage of masking, such as happiness mixed with a negative emotion, explicitly. In the present study, precise percentages are reported, along with differences between conditions.

Specifically, in the pattern analysis with a 15-second resolution, happiness increased in the last segment of the game. However, in the AUC analysis with nearly 45 seconds, this increase was statistically significant but disappeared with the p-value correction. In this context, happiness could potentially serve as a protective mechanism against feelings of failure and abandonment, especially in certain patients. The expression of happiness during inclusion, being positively correlated with more disgust during exclusion, could be an indicator of proneness toward social involvement and a marker of lower difficulty in emotion regulation. Drawing on insights from Gunderson and Lyons-Ruth (36), further research could investigate whether these variables indeed serve as markers that inform therapeutic decisions.

Moreover, sadness was the most commonly expressed emotion measured during both conditions. In our study, patients with BPD exhibited a high comorbidity of MDD and were medicated, suggesting that the elevated levels of sadness observed may reflect this comorbidity. Interestingly, in one study, researchers found reduced facial reactivity in BPD patients, as measured by electromyography, when recognizing facial expressions, despite their explicit reports of stronger subjective responses to negative emotions (37). Additionally, (3) also observed more negative emotions expressed in the groups of patients with BPD compared to healthy controls, although no differences were observed in the overall measurements of emotions between experimental conditions.

The discovery of the relatively high frequency of sadness throughout the entire experimental conditions, and the positive correlation between the intensity of sadness and feeling more ostracized during exclusion along with the observation that some profiles displayed anger during inclusion but decreased in the third segment during social exclusion, is a novel finding not previously reported. Similarly, the correlation observed between sadness during the condition of exclusion and the NTS score might signify patients seeking support during moments of exclusion. Some of these interactions may appear intuitively expected and thus warrant thorough investigation in future research.

Several studies have confirmed that even after social inclusion, patients with BPD report feelings of exclusion (3, 7). In our study, patients expressed more feelings of being excluded during the exclusion phase than during inclusion (NTS). This could be attributed to the fact that they were more severely ostracized than in previous studies, where the percentage of exclusion was lower.

In mental disorders such as post-traumatic stress disorder following an earthquake (38) and eating disorders (39), computerized analyses of facial expression have been useful for evaluating reactivity. Statistical analyses of outputs from the FaceReader system vary widely across the literature. Different approaches, such as linear regression (38), normalization of data with arcsine transformation followed by mixed-effects linear modeling, and correlations between scores (39), or calculation of means and standard deviations for each emotion (40), have been employed. In contrast, this study utilized behavioral data as outputs, employing various alternative processing methods such as blending and masking of emotions at both individual and group levels to compare conditions and timings. While pattern analysis offers a more detailed and illustrative follow-up of patients based on percentages, AUC analysis considers significant increases, as shown in the experimental protocol, see Figure 1. In this sense, in the best-case scenario, the pattern, and AUC analyses derived from the FaceReader software could be used hereafter to extract the principal features of the automatic facial expression during the Cyberball paradigm. Noteworthy, the implementation of shuffled analysis has shed light upon the differences between actual natural patterns of facial expressions and random ones (41).

The emotions expressed by patients with BPD, as revealed through interviews, have been associated with treatment outcomes (42–44). By employing more precise temporal tracking of the expressions of patients with BPD during both conditions, one of the aims of the proposed analyses was to gather and integrate various data acquired from the patient, including interviews, self-reports, and computer-based techniques, to identify more accurate diagnoses and consequently, predictors of individualized therapies to determine probable prognosis (45).

5 Limitations

Due to the small sample size and/or the heterogeneity of BPD pathology related to a well-known concept in BPD literature or the degree of rejection sensitivity, see Gunderson and Lyons-Ruth (36, 46), in the group analysis, there were no significant differences between conditions when corrections were made. The virtual players in this study are simulated entities, not actual individuals. Additionally, the sample exclusively consisted of women, which may limit the generalizability of the findings. Furthermore, comparisons with control subjects and other psychiatric disorders were not conducted. Neutral faces were also excluded from the analyses. It is worth noting that this research commenced prior to the onset of the COVID-19 pandemic, and therefore, the mandate for wearing masks in enclosed spaces in Mexico during subsequent years limited the scope of this study.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Comité de Ética en Investigación del INPRFM. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

IA-d-M: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Writing – review & editing. AR-S: Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Software, Writing – original draft. AR-L: Formal analysis, Methodology, Resources, Visualization, Writing – original draft, Writing – review & editing. M-LE-D: Formal analysis, Methodology, Resources, Visualization, Writing – review & editing. EV-M: Formal analysis, Resources, Visualization, Writing – review & editing. AR-D: Conceptualization, Methodology, Software, Supervision, Writing – review & editing. JM-D: Conceptualization, Writing – review & editing. IV-M: Methodology, Software, Writing – review & editing. ER-T: Methodology, Conceptualization, Investigation, Visualization, Software, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. The FaceReader software was purchased using funds from the Consejo Nacional de Humanidades, Ciencia y Tecnología (CONAHCyT) (National Council of Humanities, Science, and Technology). AR-S was granted a scholarship to serve her psychiatric residency at the INPRFM. The Cátedras-CONACyT program 1080, with the project “Multidisciplinary and Integral Management of Borderline Personality Disorder and associated comorbidities”, financed AR-L. Leslie Calderón-Guzmán received the “Verano de la Investigación Científica” (Summer Scientific Research) scholarship in 2020, granted by the “Academia Mexicana de las Ciencias” (Mexican Academy of Sciences), to assist in the writing of this paper.

Acknowledgments

The authors would like to express their gratitude to the medical residents who assisted in evaluating the patients. Special thanks to Marco Rosetti for providing his camera and assisting with the recordings. Additionally, we appreciate Reyna López-Blé for reviewing the English language of a previous version of the manuscript.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Matzke B, Herpertz SC, Berger C, Fleischer M, Domes G. Facial reactions during emotion recognition in borderline personality disorder: a facial electromyography study. Psychopathology. (2014) 47:101–10. doi: 10.1159/000351122

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Domes G, Czieschnek D, Weidler F, Berger C, Fast K, Herpertz SC. Recognition of facial affect in borderline personality disorder. J Pers Disord. (2008) 22:135–47. doi: 10.1521/pedi.2008.22.2.135

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Staebler K, Renneberg B, Stopsack M, Fiedler P, Weiler M, Roepke S. Facial emotional expression in reaction to social exclusion in borderline personality disorder. Psychol Med. (2011) 41:1929–38. doi: 10.1017/S0033291711000080

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Williams KD. Ostracism: A temporal need-threat model. Adv Exp Soc Psychol. (2009) 41:275–314. doi: 10.1016/S0065-2601(08)00406-1

CrossRef Full Text | Google Scholar

5. Noldus. FaceReader - Tool for automatic analysis of facial expressions. 7 edn. Wageningen, The Netherlands: Noldus Information Technology (2016).

Google Scholar

6. Chichilnisky E. A simple white noise analysis of neuronal light responses. Netw: Comput Neural Syst. (2001) 12:199. doi: 10.1080/net.12.2.199.213

CrossRef Full Text | Google Scholar

7. Gerra LM, Martina A, Silvia M, Veronica L, Paolo R, Emanuele P, et al. Autonomic vulnerability to biased perception of social inclusion in borderline personality disorder. Borderline Pers Disord Emotion Dysregul. (2021) 8:1–14. doi: 10.1186/s40479-021-00169-3

CrossRef Full Text | Google Scholar

8. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (4th ed.). American Psychiatric Publishing, Inc. (1994).

Google Scholar

9. Gorgens KA. Structured clinical interview for DSM-IV (SCID-I/SCID-II). New York, NY: Springer New York (2011). doi: 10.1007/978-0-387-79948-32011

CrossRef Full Text | Google Scholar

10. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders-IV-TR. Washington DC: American Psychiatric Association. (2000).

Google Scholar

11. Orozco-Cabal L, Rodríguez M, Herin DV, Gempeler J, Uribe M. Validity and reliability of the abbreviated barratt impulsiveness scale in spanish (bis-15s). Rev Colombiana Psiquiatría. (2010) 39:93–109. doi: 10.1016/S0034-7450(14)60239-0

CrossRef Full Text | Google Scholar

12. Pfohl B, Blum N, St. John D, McCormick B, Allen J, Black DW. Reliability and validity of the borderline evaluation of severity over time (best): A self-rated scale to measure severity and change in persons with borderline personality disorder. J Pers Disord. (2009) 23:281–93. doi: 10.1521/pedi.2009.23.3.281

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Van Beest I, Williams KD. When inclusion costs and ostracism pays, ostracism still hurts. J Pers Soc Psychol. (2006) 91:918. doi: 10.1037/0022-3514.91.5.918

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Gerber J, Chang S-H, Reimel H. Construct validity of williams’ ostracism needs threat scale. Pers Individ Dif. (2017) 115:50–3. doi: 10.1016/j.paid.2016.07.008

CrossRef Full Text | Google Scholar

15. Hervás G, Jódar R. Adaptación al castellano de la escala de dificultades en la regulación emocional. Clínica y Salud. (2008) 19:139–56.

Google Scholar

16. Noldus. FaceReader - Tool for automatic analysis of facial expressions. 6 edn. Wageningen, The Netherlands: Noldus Information Technology (2014).

Google Scholar

17. Sakai HM. White-noise analysis in neurophysiology. Physiol Rev. (1992) 72:491–505. doi: 10.1152/physrev.1992.72.2.491

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Pierce JR. An introduction to information theory: symbols, signals and noise. Estados Unidos: Dover Publications. (2012).

Google Scholar

19. Porat B. Digital processing of random signals. Theory and methods. Prentice Hall Information and System Sciences Series; Prentice Hall, Englewood Cliffs, NJ, USA (1994).

Google Scholar

20. Rodríguez EE, Hernández-Lemus E, Itzá-Ortiz BA, Jiménez I, Rudomín P. Multichannel detrended fluctuation analysis reveals synchronized patterns of spontaneous spinal activity in anesthetized cats. PloS One. (2011) 6:e26449. doi: 10.1371/journal.pone.0026449

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Zhang Z, Beck MW, Winkler DA, Huang B, Sibanda W, Goyal H, et al. Opening the black box of neural networks: methods for interpreting neural network models in clinical applications. Ann Trans Med. (2018) 6:216. doi: 10.21037/atm.2018.05.32

CrossRef Full Text | Google Scholar

22. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing (2021).

Google Scholar

23. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). Americna Psychiatric Association Publishing (2013). doi: 10.1176/appi.books.9780890425596

CrossRef Full Text | Google Scholar

24. Williams KD, Jarvis B. Cyberball: A program for use in research on interpersonal ostracism and acceptance. Behav Res Methods. (2006) 38:174–80. doi: 10.3758/BF03192765

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Hartgerink CH, Van Beest I, Wicherts JM, Williams KD. The ordinal effects of ostracism: A meta-analysis of 120 cyberball studies. PloS One. (2015) 10:e0127002. doi: 10.1371/journal.pone.0127002

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Wirth JH, Lynam DR, Williams KD. When social pain is not automatic: Personality disorder traits buffer ostracism’s immediate negative impact. J Res Pers. (2010) 44:397–401. doi: 10.1016/j.jrp.2010.03.001

CrossRef Full Text | Google Scholar

27. Meehan KB, De Panfilis C, Cain NM, Antonucci C, Soliani A, Clarkin JF, et al. Facial emotion recognition and borderline personality pathology. Psychiatry Res. (2017) 255:347–54. doi: 10.1016/j.psychres.2017.05.042

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Villanueva-Valle J, Díaz J-L, Jiménez S, Rodríguez-Delgado A, Arango de Montis I, León-Bernal A, et al. Facial and vocal expressions during clinical interviews suggest an emotional modulation paradox in borderline personality disorder: An explorative study. Front Psychiatry. (2021) 12:628397. doi: 10.3389/fpsyt.2021.628397

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Culicetto L, Ferraioli F, Lucifora C, Falzone A, Martino G, Craparo G, et al. Disgust as a transdiagnostic index of mental illness: A narrative review of clinical populations. Bull Menninger Clinic. (2023) 87:53–91. doi: 10.1521/bumc.2023.87.suppA.53

CrossRef Full Text | Google Scholar

30. Schienle A, Haas-Krammer A, Schöggl H, Kapfhammer H-P, Ille R. Altered state and trait disgust in borderline personality disorder. J Nervous Ment Dis. (2013) 201:105–8. doi: 10.1097/NMD.0b013e31827f64da

CrossRef Full Text | Google Scholar

31. Kot E, Kostecka B, Radoszewska J, Kucharska K. Self-disgust in patients with borderline personality disorder. the associations with alexithymia, emotion dysregulation, and comorbid psychopathology. Borderline Pers Disord Emotion Dysregul. (2023) 10:24. doi: 10.1186/s40479-023-00232-1

CrossRef Full Text | Google Scholar

32. Unoka Z, Fogd D, Füzy M, Csukly G. Misreading the facial signs: Specific impairments and error patterns in recognition of facial emotions with negative valence in borderline personality disorder. Psychiatry Res. (2011) 189:419–25. doi: 10.1016/j.psychres.2011.02.010

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Dudas RB, Mole TB, Morris LS, Denman C, Hill E, Szalma B, et al. Amygdala and dlpfc abnormalities, with aberrant connectivity and habituation in response to emotional stimuli in females with bpd. J Affect Disord. (2017) 208:460–6. doi: 10.1016/j.jad.2016.10.043

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Reyes Soto A. Frecuencia e intensidad de expresiones faciales de emociones negativas en pacientes con trastorno límite de la personalidad durante una experiencia de exclusión social: relación con gravedad, impulsividad, desregulación emocional, desconfianza y conflictos interpersonales. Universidad Nacional Autónoma de México (2020). Available at: http://132.248.9.195/ptd2021/febrero/0807820/Index.html. Dissertation/Psychiatry Residency’s thesis.

Google Scholar

35. Rosales Lagarde A, Rodríguez Torres EE, Vázquez Mendoza E, Eraña Díaz ML, Reyes Soto A, Arango de Montis I. Diferencias entre las emociones medidas por el facereader durante el cyberball de acuerdo al nivel de desconfianza en pacientes con trastorno límite de la personalidad. Rev Bio Cienc. (2020) 7:111. doi: 10.15741/revbio.07Suppl.e1097. LXIII Congreso Nacional de la Sociedad Mexicana de Ciencias Fisiológicas, A.C.

CrossRef Full Text | Google Scholar

36. Gunderson JG, Lyons-Ruth K. Bpd’s interpersonal hypersensitivity phenotype: A gene-environment-developmental model. J Pers Disord. (2008) 22:22. doi: 10.1521/pedi.2008.22.1.22

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Steinbrenner A, Bertsch K, Berger C, Krauch M, Spieß K, Kleindienst N, et al. Decreased facial reactivity and mirroring in women with borderline personality disorder - a facial electromyography study. Psychiatry Res Commun. (2022) 2:100040. doi: 10.1016/j.psycom.2022.100040

CrossRef Full Text | Google Scholar

38. Fujiwara T, Mizuki R, Miki T, Chemtob C. Association between facial expression and ptsd symptoms among young children exposed to the great east Japan earthquake: a pilot study. Front Psychol. (2015) 6:1534. doi: 10.3389/fpsyg.2015.01534

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Leppanen J, Dapelo MM, Davies H, Lang K, Treasure J, Tchanturia K. Computerised analysis of facial emotion expression in eating disorders. PloS One. (2017) 12:e0178972. doi: 10.1371/journal.pone.0178972

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Sato W, Hyniewska S, Minemoto K, Yoshikawa S. Facial expressions of basic emotions in Japanese laypeople. Front Psychol. (2019) 10:259. doi: 10.3389/fpsyg.2019.00259

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Sakai HM, Ken-Ichi N, Korenberg MJ. White-noise analysis in visual neuroscience. Visual Neurosci. (1988) 1:287–96. doi: 10.1017/S0952523800001942

CrossRef Full Text | Google Scholar

42. Arango de Montis I, Miranda E, Ferrer JCS, Fresán A, Ortega MAR, Vargas AN, et al. Changes in facial emotion expression during a psychotherapeutic intervention for patients with borderline personality disorder. J Psychiatr Res. (2019) 114:126–32. doi: 10.1016/j.jpsychires.2019.04.026

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Dammann G, Rudaz M, Benecke C, Riemenschneider A, Walter M, Pfaltz MC, et al. Facial affective behavior in borderline personality disorder indicating two different clusters and their influence on inpatient treatment outcome: A preliminary study. Front Psychol. (2020) 11:1658. doi: 10.3389/fpsyg.2020.01658

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Sallin L, Geissbüehler I, Grandjean L, Beuchat H, Martin-Soelch C, Pascual-Leone A, et al. Self-contempt, the working alliance and outcome in treatments for borderline personality disorder: An exploratory study. Psychother Res. (2021) 31:765–77. doi: 10.1080/10503307.2020.1849848

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Bartkiene E, Steibliene V, Adomaitiene V, Lele V, Cernauskas D, Zadeike D, et al. The perspectives associated with the computer-based diagnostic method of depressive disorder. Front Psychiatry. (2018) 9:687. doi: 10.3389/fpsyt.2018.00687

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Gratz KL, Rosenthal MZ, Tull MT, Lejuez C, Gunderson JG. An experimental investigation of emotion dysregulation in borderline personality disorder. J Abnormal Psychol. (2006) 115:850. doi: 10.1037/0021-843X.115.4.850

CrossRef Full Text | Google Scholar