Daniela A. Espinoza Oyarce
Richard Burns
Peter Butterworth
Nicolas Cherbuin- Centre for Research on Ageing, Health & Welbeing, Research School of Population Health, The Australian National University, Canberra, ACT, Australia
The reinforcement sensitivity theory (RST) proposes that neurobiological systems mediate protective and appetitive behaviours and the functioning of these systems is associated to personality traits. In this manner, the RST is a link between neuroscience, behaviour, and personality. The theory evolved to the present revised version describing three systems: fight-flight-freezing, behavioural approach/activation (BAS), and behavioural inhibition (BIS). However, the most widely available measure of the theory, the BIS/BAS scales, only investigates two systems. Using a large longitudinal community survey, we found that the BIS/BAS scales can be re-structured to investigate the three systems of the theory with a BIS scale, three BAS scales, and a separate fight-flight-freezing system (FFFS) scale. The re-structured scales were age, sex, and longitudinally invariant, and associations with personality and mental health measures followed theoretical expectations and previously published associations. The proposed framework can be used to investigate behavioural choices influencing physical and mental health and bridge historical with contemporary research.
Introduction
The reinforcement sensitivity theory (RST) is one of the most influential behavioural models of the 20th century. The RST was developed from empirical data on the neurobiology of anxiety (Gray, 1970, 1982; Gray and McNaughton, 2000) and provides a unique link between brain and behaviour. Classical RST describes behaviour as the interrelation of three systems: the behavioural inhibition (BIS), behavioural approach/activation (BAS), and fight-flight (FFS). The BIS includes the hippocampus and closely connected structures and is a global avoidance system that inhibits ongoing behaviour in novel, uncertain or dangerous conditions to increase risk assessment and attention. Punishment and fear activate this system and BIS functioning is associated with anxiety (Gray, 1970, 1982; Corr et al., 1995; Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006b). The BAS comprises reward pathways and is a global approach system that directs behaviour towards a goal or reward, increasing exploration to ensure biological needs are met. Consequently, rewards activate this system and BAS functioning is associated with impulsivity (Corr et al., 1995; Pickering and Gray, 1999; Gray and McNaughton, 2000; Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006b). Finally, the FFS includes midbrain structures and is a secondary avoidance system that mediates fight-or-flight behaviour. Pain activates this system and FFS functioning is associated with anger and panic (Corr et al., 1995; Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006b) but these associations were less clear than for the BIS and BAS (Carver and White, 1994; Smillie et al., 2006b). From the descriptions above, the link between brain and behaviour also extends to personality and, in this manner, the RST is a model that investigates current and future behaviour and can be utilised to understand physical and mental health outcomes.
Considering the potential of the RST, a number of questionnaires were developed to estimate BIS and BAS sensitivities. The most widely used measure are the BIS/BAS scales (Carver and White, 1994), which have been used to investigate sensitivities to these systems and to demonstrate substantive associations to personality and health. For instance, the BAS has been shown to be positively associated to impulsivity (Poythress et al., 2008; Beck et al., 2009) and positive affect (Jorm et al., 1999; Smillie et al., 2006a), and impulsivity-related behaviours such as aggression and delinquency (Yu et al., 2011), smoking uptake (Baumann et al., 2014), drug and alcohol use (Johnson et al., 2003; Wardell et al., 2011), and binge/purge eating disorders (Beck et al., 2009). On the other hand, the BIS has been shown to be positively associated to neuroticism (Jorm et al., 1999; Heym et al., 2008; Yu et al., 2011) and negative affect (Jorm et al., 1999), and neuroticism-related disorders such as anxiety (Jorm et al., 1999; Johnson et al., 2003; Poythress et al., 2008) and depression (Jorm et al., 1999; Johnson et al., 2003). Emerging data on fear, anxiety, and reward behaviour, however, led to a major revision of the RST (Gray and McNaughton, 2000).
In the revised RST, the BIS and BAS are reconceptualised and FFS redefined into the fight-flight-freezing system (FFFS). The revised BIS is a supervisory system that resolves conflicts when approach and avoidance behaviours are concurrently engaged, or when competing goals or competing avoidance actions are possible. The system inhibits ongoing behaviour during conflicts and increases risk assessment, attention, and cautious approach to promote a single response. BIS activation during conflict resolution is associated with anxiety with inability to resolve conflicts associated with increased levels of anxiety (Gray and McNaughton, 2000; Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006b). The revised BAS remains a global approach system but BAS functioning is multidimensional and associated with both impulsivity and extraversion (Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006a,b). Finally, the FFFS is a global avoidance and escape system in threatening conditions and promotes fight, flight, or freezing behaviour depending on perceived risk. Specifically, high risk promotes attack, intermediate risk promotes flight or freezing, and low risk does not engage defensive behaviour. FFFS functioning is associated with fear and panic (Gray and McNaughton, 2000; Corr, 2004; Corr and Perkins, 2006; Smillie et al., 2006b).
The revision of the RST, however, was not supplemented with a concurrent update of psychometric tools. The first measure reflecting the new conceptualisation was developed 9 years after the revision of the theory with the Jackson-5 scales (Jackson, 2009), yet the constructs investigated did not follow theoretical associations and likely limited its use. Specifically, the BIS and BAS scales correlated positively despite the differing functions of these systems and the FFFS constructs did not converge into a single scale (Krupić et al., 2016a). A similar situation was observed for another measure, the RST questionnaire (Smederevac et al., 2014), and FFFS constructs. The recently developed revised RST questionnaire (Reuter et al., 2015) and RST personality questionnaire (Corr and Cooper, 2016) have addressed the methodological issues of the earlier measures, nevertheless there is no consensus on whether the BAS should be investigated using a single scale, as proposed by Smederevac et al. (2014) and Reuter et al. (2015), or multiple subscales, as proposed by Corr and Cooper (2016) and Krupić et al. (2016a,b). In this manner, newer measures are yet to be widely adopted and there is currently no robust way to link past research based on the BIS/BAS scales, which have been used for over 25 years and provide a wealth of behavioural and personality data. Indeed, previous attempts at linking past research with the BIS/BAS scales (Krupić et al., 2016a) did not address the conflation of fear and anxiety constructs in the BIS scale, which are based on the original conceptualisation.
Therefore, the aims of the present study were to investigate whether the earlier conceptualisation of the BIS based on the scales of Carver and White (1994), which includes items relating to both anxiety and fear, can also be used to additionally estimate the fear component of the FFFS. In addition, the study will also systematically investigate the stability of the BIS, BAS, and FFFS scales across different cohorts in terms of age, sex, and time using rich epidemiological data.
Materials and Methods
Participants and Procedure
Participants were drawn from the Personality and Total Health (PATH) Through Life Project, a longitudinal community-based sample from the cities of Canberra and neighbouring Queanbeyan, Australia (Anstey et al., 2012). Participants were randomly selected from these communities using electoral rolls, since voting is compulsory for all Australian citizens aged 18 and above. PATH included 7,485 participants at baseline, within three age cohorts: young (20–24 years), midlife (40–44 years), and old-age (60–64 years).
All participants provided written consent and were interviewed every 4 years either at The Australian National University or at their home, where they completed self-report questionnaires using personal computers (Anstey et al., 2012; Windsor et al., 2012). Each wave of data collection of the PATH project has been approved by the Human Research Ethics Committee of The Australian National University. The present study includes data from four waves of measurement from 1999 to 2001, 2003 to 2005, 2007 to 2010, and 2011 to 2014 (Table 1). Retention rates from baseline were 53% for young, 71% for midlife, and 64% for old-age cohorts; with a higher proportion of females in the young cohort and higher proportion of males in the old-age cohort after 12 years of follow-up (Anstey et al., 2021).
Table 1. Descriptive statistics by age and wave of measurement.
Measures
Behavioural Inhibition and Behavioural Activation System Scales
The BIS/BAS scales (Carver and White, 1994) comprise 20 questions structured in a four-point Likert scale, with responses ranging from very false for me to very true for me. The additional four filler questions were not included in analyses. The BIS scale includes seven items assessing inhibitory behaviour and sensitivity to punishment, with items reflecting responses to potentially unpleasant scenarios. The BAS scale consists of three subscales. The drive subscale (BAS-d) includes four items reflecting the pursuit of goals; the fun-seeking subscale (BAS-f) includes four items reflecting willingness to approach rewarding situations; and the reward responsiveness subscale (BAS-r) includes five items reflecting positive responses to rewards.
Revised Eysenck Personality Questionnaire Short-Scale
The revised Eysenck personality questionnaire short-scale (EPQ-R; Eysenck et al., 1985), comprises 36 questions structured with dichotomous responses yes and no for three scales. The lie scale was not included in PATH. The extraversion scale includes 12 items addressing attitudes towards social interactions and external arousal; the neuroticism scale includes 12 items addressing attitudes towards feelings of worry and emotional insecurity; and the psychoticism scale includes 12 items addressing attitudes towards hostility and non-conformism.
Positive and Negative Affect Schedule
The positive and negative affect schedule (PANAS; Watson et al., 1988), comprises 20 questions in a five-point Likert scale with responses ranging from very slightly or not at all to extremely. The positive affect scale includes 10 descriptors assessing enthusiasm and alertness, and the negative affect scale includes 10 descriptors assessing distress and unpleasant states within the past 4 weeks.
Goldberg Anxiety and Depression Scales
The Goldberg anxiety and depression scales (GADS; Goldberg et al., 1988), comprise 18 questions structured with dichotomous responses yes and no. The anxiety scale includes nine questions assessing generalised anxiety disorder symptoms, and the depression scale includes nine questions assessing major depressive disorder symptoms within the past 4 weeks. The first four questions of each scale address core disorder symptoms, while the remaining questions address supplementary symptoms associated with physical disturbances. Scores of at least seven in the anxiety scale represent optimal screening values for generalised anxiety, and scores of at least five in the depression scale represent optimal screening values for major depressive disorder (Kiely and Butterworth, 2015).
Statistical Analysis
R version 3.6.0 (R Core Team, 2019) for Windows was used for all statistical analyses. Exploratory factor analyses (EFAs) were conducted using the psych package version 1.8.12 (Revelle, 2018). Confirmatory factor analyses (CFAs) were conducted using the lavaan package version 0.6-3 (Rosseel, 2012; see Supplement for R code). A p ≤ 0.001 was considered significant for all analyses and our sample met size recommendations for factor analyses (Mundfrom et al., 2005; Li, 2016).
Exploratory Factor Analyses
Exploratory factor analyses were conducted to investigate the latent factor structure of the BIS/BAS scales. EFAs were conducted in a random subsample representing 60% of PATH wave 1 data, while the remaining 40% of the data was reserved for subsequent CFAs. There were no significant differences between EFA and CFA subsamples in terms of age, sex, education, or BIS/BAS scores (Supplementary Table 1). EFAs were estimated with weighted least squares (WLS) extraction owing to the distribution of the manifest item indicators, with an oblique Oblimin rotation to allow for possible correlations between factors, based on raw scores from a polychoric correlation matrix (Revelle, 2018). Several EFA models were investigated starting with a two-factor solution representing BIS and BAS as single general scales, and systematically increasing factor number until appropriate model fit was established. To ensure that the identified model represented the best fit for the data, a model with one extra factor was further investigated (Fabrigar and Wegener, 2012). The total number of factors examined was six.
Multiple-Groups Confirmatory Factor Analyses
Multiple-groups CFAs were used to assess measurement invariance (Jöreskog, 1971) of the EFA model. Age and sex invariance were assessed in the remaining 40% subsample of PATH wave 1 data, while longitudinal invariance was conducted with all waves. CFAs were conducted with diagonal WLS extraction allowing correlations between factors, based on raw scores from polychoric covariance matrices (Rosseel, 2012). This method has been shown to outperform robust maximum likelihood estimation when dealing with Likert-type scales with four response categories (Li, 2016). For model identification, the first variable of each latent factor was set to 1, latent factor variances were freely estimated, and intercepts were set to 0.
Measurement invariance was investigated using the method proposed by Wu and Estabrook (2016) for ordinal categorical data since previous methods, i.e., (Meredith, 1993), are based on continuous data. Briefly, this approach recognises the parameter constraints required for model identification as invariance constraints and imposes more stringent model parameters in a series of nested models. First, configural invariance is assessed with free estimation of thresholds and factor loadings within groups to ensure suitability of the model structure, with residual variances set to 1 and factor means set to 0 for all groups. Second, scalar invariance is assessed by fixing thresholds to be equal among groups, with residual variances set to 1 and factor means set to 0 in a reference group, and estimated in the remaining groups. Finally, metric invariance is investigated by fixing both thresholds and factor loadings to be equal among groups, with residual variances set to 1 and factor means set to 0 in the same reference group as scalar invariance, and estimated in the remaining groups (Supplementary Table 2). At each step, model fit is assessed. Compared to other methodologies (Meredith, 1993; Millsap and Yun-Tein, 2004), the testing order is reversed since it was demonstrated that metric invariance can only be tested if prior scalar invariance is established (Wu and Estabrook, 2016).
Model Fit
Several goodness of fit indicators were used to assess EFA and CFA models. The χ2 ratio test is sensitive to sample size (Bentler and Bonett, 1980; Windsor et al., 2012; Svetina et al., 2019) and small deviations in likelihood would reach statistical significance. Therefore, we included Tucker-Lewis index (TLI), comparative fit index (CFI), root mean square error of approximation (RMSEA), and Bayesian information criterion (BIC) as indicators. We considered a TLI/CFI ≥ 0.90, RMSEA ≤ 0.08, and the smallest BIC values as indicators of good fit. These threshold values are close to previous recommendations for normally distributed data (Hu and Bentler, 1999; Poythress et al., 2008; Fabrigar and Wegener, 2012).
Measurement Invariance
Measurement invariance is established when equality constraints yield appropriate model fit and there is a non-significant χ2 test between scalar and metric nested models (Wu and Estabrook, 2016; Svetina et al., 2019). However, the χ2 test is likely to reject invariant models with large number of groups and/or factors (Svetina and Rutkowski, 2017; Svetina et al., 2019). Therefore, we additionally considered a change in RMSEA (ΔRMSEA) ≤ 0.01 and change in CFI (ΔCFI) ≥ −0.002 between the two most restricted models as indicators of invariance (Svetina and Rutkowski, 2017).
Scale Correlations
Associations between BIS/BAS scales and EPQ-R, PANAS, and GADS were investigated in PATH wave 1 given the larger sample and data available. A CFA was conducted to obtain latent scores of the identified RST systems. Items 1 and 6 from the BIS scale were reverse coded to represent psychometrically meaningful associations with other scales.
Based on theoretical associations and previous literature, we expected the BIS scale to be positively associated to neuroticism, negative affect, and anxiety (Carver and White, 1994; Jorm et al., 1999; Gray and McNaughton, 2000; Johnson et al., 2003; Heym et al., 2008; Poythress et al., 2008; Jackson, 2009; Yu et al., 2011; Kramer and Rodriguez, 2018); and negatively associated to psychoticism (Jorm et al., 1999; Heym et al., 2008). We also expected all BAS subscales to be positively associated to extraversion and positive affect (Carver and White, 1994; Jorm et al., 1999; Gray and McNaughton, 2000; Smillie et al., 2006a; Heym et al., 2008; Beck et al., 2009; Kramer and Rodriguez, 2018), with BAS-f additionally positively associated to psychoticism (Jorm et al., 1999; Smillie et al., 2006a; Heym et al., 2008); and negatively associated to neuroticism and negative affect. We also expected a negative correlation between BAS and depression (Gray and McNaughton, 2000); however, a positive association between BIS and depression is likely given previous studies (Jorm et al., 1999; Johnson et al., 2003). Finally, we expected an FFFS scale to be positively associated to neuroticism and negative affect, and negatively associated to psychoticism based on previous literature (Heym et al., 2008; Kramer and Rodriguez, 2018).
Results
Latent Structure of the BIS/BAS Scales
We first investigated the latent factor structure of the BIS/BAS scales. The PATH wave 1 EFA subsample included 4,491 participants [mean age = 43.38 (SD = 16.40) years, 50.8% female]. All scales showed acceptable internal consistency based on pre-established items (α ≥ 0.7; Supplementary Table 1).
A five-factor model best fitted the data, accounting for 52% of the variance. Addition of a sixth factor worsened fit indices, and therefore was not retained (Table 2; see Supplementary Figures 1, 2 for four- and six-factor models, respectively). The loading structure of the five-factor model showed BAS items divided into three factors representing each of the established BAS subscales, and BIS items divided into two factors with items 1 and 6 loading separately (Figure 1; Supplementary Table 3). Considering that items 1 (“I have very few fears compared to my friends”) and 6 (“even when something bad is about to happen to me, I rarely experience fear or nervousness”) are associated with fearful rather than anxious emotions, the fifth factor was identified as the FFFS.
Table 2. Exploratory factor analysis of the BIS/BAS scales.
Figure 1. Behavioural inhibition system scale BIS/BAS/FFFS five-factor model. Loadings < 0.32 have been omitted. Dashed lines represent lower double loadings. BIS, behavioural inhibition system; BAS, behavioural approach/activation system; BAS-d, drive; BAS-f, fun-seeking; BAS-r, reward responsiveness; FFFS, fight-flight-freezing; χ2, Chi-square; TLI, Tucker Lewis index; RMSEA, root mean square error of approximation; CI, confidence interval; and BIC, bayesian information criterion.
Positive factor correlations (r ≥ 0.41) were found between the three BAS subscales, BIS and FFFS (r = 0.45), and BIS and BAS-r (r = 0.31).
Measurement Invariance
Age and Sex Invariance
A series of increasingly restrictive nested models were conducted to establish invariance of the structure and assess the generalisability of our findings. The PATH wave 1 CFA subsample included 2,994 participants [mean age = 43.60 (SD = 16.14) years, 50.2% female; Supplementary Table 1]. Configural, scalar, and metric nested models showed appropriate fit indices for age and sex (Table 3; see Supplementary Tables 4, 5 for factor loadings and thresholds for fully invariant models, respectively). A comparison of the most restrictive models yielded significant χ2 tests as expected; however, ΔRMSEAs and ΔCFIs were below the established cut-offs indicating age and sex invariance.
Table 3. Measurement invariance models of the BIS/BAS/FFFS scales.
In the invariant models, the pattern of factor correlations was similar between cohorts but the magnitude varied by age and sex. Positive correlations between BIS and BAS-r were highest in old-age, with decreasing values as age decreased. Positive correlations between BIS and FFFS were highest in the young, with decreasing values as age increased. Positive correlations between BAS scales were highest in old-age and lowest in midlife; while negative correlations between BAS-d and BAS-f, and FFFS highest in the young, and lowest in midlife. Finally, positive correlations between BIS and BAS-r, and BIS and FFFS were highest in females; while positive correlations between BAS subscales, and negative correlations between BAS-d and BAS-r, and FFFS were highest in males (Supplementary Table 6).
Longitudinal Invariance
Configural, scalar, and metric nested models showed appropriate fit indices and were comparatively better than those for age and sex invariance (Table 3; see Supplementary Tables 4, 5 for factor loadings and thresholds for fully invariant models, respectively). Comparison of the most restrictive models yielded a significant χ2 test again, but ΔRMSEA and ΔCFI were below the established cut-offs indicating longitudinal invariance.
The pattern of moderate to high correlations in the longitudinal invariant model was similar between waves. There was low variability in the magnitude of these correlations when compared to the age and sex invariant models. Correlations between BAS-d and BAS-f showed the highest variability between waves (range = 0.64–0.72; Supplementary Table 7).
Scale Correlations
The internal consistencies of extraversion, neuroticism, PANAS, and GADS scales were high (α ≥ 0.78); while the internal consistency of the psychoticism scale was poor (α = 0.48; Table 4). Most scales investigated showed significant sex differences (Mann-Whitney-Wilcoxon test, p < 0.001); however, correlations between scales did not differ substantially by sex and were thus investigated in the whole sample (Supplementary Figures 3, 4).
Table 4. Descriptive statistics of personality and mental health measures at wave 1.
Significant positive correlations were found between BIS and neuroticism, negative affect, and anxiety and depression; between BAS subscales and extraversion, and positive affect; and between BAS-f and psychoticism. The magnitude of these correlations varied from r = 0.57 for BIS and neuroticism, to r = 0.22 for BAS-r and positive affect. A significant moderate negative correlation between BIS and psychoticism, and a significant moderate positive correlation between BAS-d and psychoticism were also found (Figure 2).
Figure 2. Associations between BIS/BAS/FFFS, EPQ-R, PANAS, and GADS at wave 1. Far right bar shows value of correlation. BIS, behavioural inhibition system; BAS, behavioural approach/activation system; BAS_d, drive; BAS_f, fun-seeking; BAS_r, reward responsiveness; FFFS, fight-flight-freezing; EPQ, Eysenck personality questionnaire; EPQ_E, extraversion; EPQ_N, neuroticism; EPQ_P, psychoticism; PAS, positive affect; NAS, negative affect; GAS, Goldberg anxiety; and GDS, Goldberg depression. *p ≤ 0.05, **p ≤ 0.001, and ***p ≤ 0.0001.
The pattern of correlations between FFFS and other scales largely resembled that of the BIS, except that positive correlations were greater with BIS and negative correlations were greater with FFFS. Significant positive correlations were found between FFFS and neuroticism, negative affect, and anxiety and depression; while significant negative correlations were found between FFFS and extraversion, psychoticism, and positive affect (Figure 2).
Discussion
The present work contributes important data to behavioural and personality research. To the best of our knowledge, this is the first study to validate the BIS/BAS scales and investigate longitudinal invariance in a large community sample using contemporary methods that accurately analyse questionnaire data. Indeed, the methodology followed here was guided by methods and recommendations developed within the last 5 years (Li, 2016; Wu and Estabrook, 2016; Svetina and Rutkowski, 2017; Edossa et al., 2018; Svetina et al., 2019). The main finding of the study was that the BIS/BAS scales can effectively identify the three systems of the revised reinforcement sensitivity theory in a five-factor structure comprising the BIS, three BAS subscales, and the fear construct of the FFFS. Notably, this structure was found to be stable by age and sex, and, remarkably, within individuals over a period of 12 years.
The BIS/BAS/FFFS Scales
The investigation of the factor structure of the BIS/BAS scales was primarily exploratory to identify the most parsimonious configuration of the data. We identified a BIS scale composed of five items instead of seven expected from the original BIS scale; three BAS subscales in line with the original BAS configuration; and a separate factor identified as the FFFS with items 1 and 6 from the original BIS scale. Consistent with this structure, items 1 and 6 have been previously shown to be distinct from the BIS scale (Johnson et al., 2003;Poythress et al., 2008; Beck et al., 2009) and measurement invariance demonstrated the stability of the BIS/BAS/FFFS scales.
This latter finding has direct implications for the FFFS scale, which includes the only two reverse-coded items and its separation from the BIS could result from a methodological artifact. We demonstrated using stringent analyses that a major component of the revised theory, the fear construct of the FFFS, is present, measureable and a robust construct to age, sex, and time stratification. Additionally, divergent associations between FFFS and BIS scales and other personality measures provide evidence of construct validity. Specifically, only FFFS items have been shown to associate with harm avoidance, while BIS items do not (Poythress et al., 2008); and only BIS items associate with BAS reward responsiveness, while FFFS items do not (Poythress et al., 2008; Beck et al., 2009). In this manner, the FFFS scale is linked to protective behaviours in line with the revised RST.
In contrast to this structure, a number of studies have proposed alternative configurations for the BIS/BAS scales. However, we consider that these results arise because of differences in analytical and methodological approaches. First with respect to analytical approach, item 4 from the original BIS scale (“if I think something unpleasant is going to happen I usually get pretty ‘worked-up’”) was additionally identified as part of the FFFS when the BIS scale was investigated in isolation (Heym et al., 2008). Considering that the scales represent RST systems that interact with one another, the investigation of latent constructs should follow a holistic approach, where the entirety of the questionnaire is assessed. Second with respect to methodology, previous studies that have supported the original four-factor structure (Heubeck et al., 1998; Jorm et al., 1999; Cooper et al., 2007), a three-factor structure without BAS fun-seeking scale (Pagliaccio et al., 2016), and a two-factor structure with a single BAS scale (Yu et al., 2011) have used CFA and/or invariance methodology designed for continuous data. Indeed, studies supporting a five-factor structure have used exploratory approaches (Johnson et al., 2003; Poythress et al., 2008) or CFA methodology designed for ordinal data (Beck et al., 2009) which are better suited for analysis.
Personality and Mental Health Associations
Most associations between the BIS, BAS, and FFFS scales and personality and mental health measures followed theoretical expectations (Gray, 1970, 1982; Corr et al., 1995; Pickering and Gray, 1999; Gray and McNaughton, 2000; Corr, 2004), such as BIS and FFFS and negative affect, and BAS and extraversion. However, the reported associations between BIS and depression, FFFS and anxiety and depression, and BAS drive and psychoticism were unexpected theoretically (Figure 2).
The positive correlation between BIS and depression has been reported in literature (Jorm et al., 1999; Johnson et al., 2003; Yu et al., 2011), but it is unexpected since depression should reflect low BAS activation given anhedonic symptoms (Gray and McNaughton, 2000). However, we only found low correlations between BAS subscales and depression. Considering the high rates of comorbidity between anxiety and depression (Van Tol et al., 2010; World Health Organization, 2017; Espinoza Oyarce, et al., 2020), it is likely that the association between BIS and depression is attributable to anxiety as previously suggested (Bijttebier et al., 2009) and in line with the high correlation between anxiety and depression scales reported here. Our results show that in community-dwelling populations, anxiety and depression symptoms co-exist and this co-existence likely underlies the association between BIS and depression. This may not be a feature in other populations with clinically relevant depression symptoms and there is evidence of BAS hypoactivation in patients with ahedonic depressive symptoms, but not in patients with mixed anxiety and depression symptoms (Bijttebier et al., 2009). Future studies should investigate participants separately based on anxiety and depression symptoms to assess BIS and BAS sensitivities.
The unexpected positive correlations between FFFS and anxiety and depression may also result from underlying associations. Indeed, the FFFS is theoretically linked to neuroticism (Corr, 2004), which in turn is associated with both anxiety and depression. Our results show that these associations also extend to the fear component of the FFFS; however, similar to our discussion above, future research should investigate whether these associations hold in clinical populations, particularly those with panic and phobia symptoms. How the fear component of the FFFS relates to other aspects of human behaviour, such as health, diet, and life choices, remains the focus of future research.
Finally, the correlation between BAS drive subscale and psychoticism may require further investigation. This association has been reported in adults with the short-scale (Jorm et al., 1999) but not in undergraduate students with the full-scale (Smillie et al., 2006a; Heym et al., 2008). Age and demographics may explain these differences but the psychoticism scale likely plays a role. The full-scale showed higher internal consistencies and future investigations with the full-scale in adults would clarify the association between BAS and psychoticism.
Implications for Personality and Behavioural Research
One of the major implications of our study is that the BIS, BAS, and FFFS scales are invariant constructs that can be used to study the RST in populations regardless of sex, age, and time between assessments. Furthermore, because the psychometric properties of these scales may reflect individual strengths and vulnerabilities to health and disease, the BIS, BAS, and FFFS scales could improve our understanding of the development of mental and physiological disorders and ultimately provide an avenue to prevent them.
For instance, individuals with high BIS scores would be predicted to be more sensitive to negative emotionality when an expected reward is not received, such as a salary raise, and more likely to have high anxiety and neuroticism. These conditions may translate in higher scores on anxiety and depression measures, and ultimately lead to clinical symptoms (Johnson et al., 2003; Poythress et al., 2008; Bijttebier et al., 2009). On the other hand, individuals with high BAS scores would be predicted to be more sensitive to positive emotionality when receiving a reward, such as a large meal after hard work, and more likely to have high extraversion and impulsivity; which may translate into higher risk of developing obesity and poorer cardiovascular health. Studies have shown that high BAS sensitivity is linked to purge/binge eating disorders (Beck et al., 2009; Bijttebier et al., 2009) and food craving (Franken and Muris, 2005), ultimately increasing obesity risk (Bijttebier et al., 2009) and development of associated metabolic diseases. There is, therefore, the need for more research aimed at identifying how BIS, BAS, and FFFS sensitivities relate to unhealthy behaviour and/or exposure to risk factors for physical and mental health. The scales here developed are suited for such investigation.
Another implication of our study is in the ongoing discussion of BAS dimensionality in behavioural research. Despite the theoretical, multidimensional nature of the BAS (Gray and McNaughton, 2000), there is currently no consensus on the dimensionality of psychometric measures of the BAS. New measures of the revised reinforcement sensitivity theory investigate BAS sensitivities using single scales, such as those proposed by Jackson (2009), Smederevac et al. (2014), and Reuter et al. (2015), and multiple scales, as proposed by Corr and Cooper (2016). Our results on the invariance of the BIS/BAS/FFFS scales agree with previous assessments that the BAS should be studied with separate scales given poor fit indices of unidimensional BAS scales (Krupić et al., 2016a). However, it is likely that the discussion of scale dimensionality will require a multimodal approach considering the neurobiological basis of the reinforcement sensitivity theory and reward circuitry, i.e., (Der-Avakian and Markou, 2011). Future studies could use the scales developed here to re-evaluate historical neuroimaging data, and neuroimaging studies comparing unidimensional and multidimensional scales will provide further insights on the psychometric dimensionality of the BAS to test the theoretical dimensionality of the system.
Limitations and Future Directions
Despite the strengths of our study in terms of methodological rigor and longitudinal data, we must acknowledge some limitations. First, the FFFS scale addresses the central fear construct of the system, but specific responses to fight, flight, and freezing behaviour may not be fully captured. Similarly, the BIS scale may not fully assess conflict resolution. Future studies investigating the FFFS and BIS with newly developed measures, such as the RST personality questionnaire (Corr and Cooper, 2016) and the revised RST questionnaire (Reuter et al., 2015), should provide clearer insight of these traits. Complementary to this, future studies investigating associations between new measures and the BIS, BAS, and FFFS scales developed here would strengthen the link between historic and contemporary research based on the revised theory. Specifically, these studies could help corroborate or update previous associations between RST systems and behavioural, psychometric, and neuroimaging data that has been collected over two decades with the BIS/BAS scales. We have conducted such analyses for psychometric data and found correspondence between the BIS/BAS/FFFS scales here reported, newer measures with the revised RST questionnaire (Reuter et al., 2015), and personality measures (Submitted).
Second, we provide evidence of age, sex, and longitudinal invariance of the BIS, BAS, and FFFS scales; however, our sample included participants aged 20 years and older. Age invariance in younger cohorts remains a topic requiring further investigation and future studies should use the methodology presented here to investigate the five-factor structure of the BIS/BAS/FFFS scales in children and adolescents and assess whether other structures based on the old theory, i.e., (Muris et al., 2005), are better suited.
Further in regards to our sample, PATH participants have higher education level and socio-economic status compared to the national Australian average since the Canberra region is more socio-economically advantaged (Anstey et al., 2012). While we did not expect significant deviations on the measures investigated here, future studies with diverse socio-economic populations will provide further validation of the BIS, BAS, and FFFS scales.
Finally, psychometric associations with the psychoticism scale should be further investigated. The revised Eysenck personality questionnaire specifically addresses low internal consistency (Eysenck et al., 1985) yet low values are still reported, i.e., α = 0.24 (Tiwari et al., 2009). The multi-faceted nature of the scale, which includes lack of empathy, impulsivity and hostility, may underlie low consistencies (Eysenck et al., 1985).
Conclusion
The Carver and White BIS/BAS scales produce robust and reliable measures that allow the investigation of the three systems of the revised reinforcement sensitivity theory. Future studies can use the framework proposed here to investigate the FFFS and BIS to assess behavioural responses linked to fear separate from those linked to anxiety. Furthermore, the validated BAS subscales will be instrumental in providing additional data to address the dimensionality of the BAS. Taken together, the BIS/BAS/FFFS scales will help build a conceptual bridge between historical and contemporary personality and behavioural research.
Data Availability Statement
The datasets presented in this article are not readily available because PATH data is not publicly available. Although the PATH study governance arrangements and the commitments made to participants do not allow for unrestricted sharing of research data, we will be happy to make available the analysed data to external parties to verify our findings or for research purposes on a case by case basis after a formal request to, and approval from, the PATH Research Committee. Requests to access the datasets in this study should be directed to Prof Nicolas Cherbuin (bmljb2xhcy5jaGVyYnVpbkBhbnUuZWR1LmF1). All other requests should be directed to aW5mb0BwYXRoc3R1ZHkub3JnLmF1.
Ethics Statement
The studies involving human participants were reviewed and approved by The Human Research Ethics Committee of The Australian National University. The participants provided their written informed consent to participate in this study.
Author Contributions
DAEO contributed to the design of the study and statistical analyses, conducted all statistical analyses in R, and managed all aspects of manuscript preparation and submission. RB contributed to the design of the study, confirmed all statistical analyses in MPlus, provided methodological input and theoretical expertise, and contributed to writing and editing of the manuscript. PB and NC contributed to the design of the study and the statistical analyses, provided methodological input and theoretical expertise, and contributed to writing and editing of the manuscript. All authors contributed to the article and approved the submitted version.
Funding
DAEO was supported by the Australian Government Research Training Program Scholarship. PATH data collection was supported by the National Health and Medical Research Council grants 973302, 179805, 418039, and 1002160 and Safe Work Australia. The funders had no role in study design, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and all authors had final responsibility for the decision to submit for publication.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
We would like to thank Anthony Jorm, Helen Christensen, Bryan Rodgers, Simon Easteal, Kaarin Anstey, Andrew Mackinnon, and the PATH Team and participants for their contribution to the research.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.737117/full#supplementary-material
Abbreviations
RST, Reinforcement sensitivity theory; BIS, Behavioural inhibition system; BAS, Behavioural approach/activation system; FFS, Fight/flight system; FFFS, Fight-flight-freezing system; EPQ-R, Revised Eysenck personality questionnaire short-scale; PANAS, Positive and negative affect schedule; GADS, Goldberg anxiety and depression scales.
References
Anstey, K. J., Butterworth, P., Christensen, H., Easteal, S., Cherbuin, N., Leach, L., et al. (2021). Cohort profile update: the PATH through life project. Int. J. Epidemiol. 50, 35–36. doi: 10.1093/ije/dyaa179
Anstey, K. J., Christensen, H., Butterworth, P., Easteal, S., Mackinnon, A., Jacomb, T., et al. (2012). Cohort profile: the PATH through life project. Int. J. Epidemiol. 41, 951–960. doi: 10.1093/ije/dyr025
Baumann, M. R., Oviatt, D., Garza, R. T., Gonzalez-Blanks, A. G., Lopez, S. G., Alexander-Delpech, P., et al. (2014). Variation in BAS-BIS profiles across categories of cigarette use. Addict. Behav. 39, 1477–1483. doi: 10.1016/j.addbeh.2014.05.028
Beck, I., Smits, D. J. M., Claes, L., Vandereycken, W., and Bijttebier, P. (2009). Psychometric evaluation of the behavioral inhibition/behavioral activation system scales and the sensitivity to punishment and sensitivity to reward questionnaire in a sample of eating disordered patients. Personal. Individ. Differ. 47, 407–412. doi: 10.1016/j.paid.2009.04.007
Bentler, P. M., and Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606. doi: 10.1037/0033-2909.88.3.588
Bijttebier, P., Beck, I., Claes, L., and Vandereycken, W. (2009). Gray’s reinforcement sensitivity theory as a framework for research on personality-psychopathology associations. Clin. Psychol. Rev. 29, 421–430. doi: 10.1016/j.cpr.2009.04.002
Carver, C. S., and White, T. L. (1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: the BIS/BAS scales. J. Pers. Soc. Psychol. 67, 319–333. doi: 10.1037/0022-3514.67.2.319
Cooper, A., Gomez, R., and Aucote, H. (2007). The behavioural inhibition system and behavioural approach system (BIS/BAS) scales: measurement and structural invariance across adults and adolescents. Personal. Individ. Differ. 43, 295–305. doi: 10.1016/j.paid.2006.11.023
Corr, P. J. (2004). Reinforcement sensitivity theory and personality. Neurosci. Biobehav. Rev. 28, 317–332. doi: 10.1016/j.neubiorev.2004.01.005
Corr, P. J., and Cooper, A. J. (2016). The reinforcement sensitivity theory of personality questionnaire (RST-PQ): development and validation. Psychol. Assess. 28, 1427–1440. doi: 10.1037/pas0000273
Corr, P. J., and Perkins, A. M. (2006). The role of theory in the psychophysiology of personality: from ivan pavlov to jeffrey gray. Int. J. Psychophysiol. 62, 367–376. doi: 10.1016/j.ijpsycho.2006.01.005
Corr, P. J., Pickering, A. D., and Gray, J. A. (1995). Personality and reinforcement in associative and instrumental learning. Personal. Individ. Differ. 19, 47–71. doi: 10.1016/0191-8869(95)00013-V
Der-Avakian, A., and Markou, A. (2011). The neurobiology of anhedonia and other reward-related deficits. Trends Neurosci. 35, 68–77. doi: 10.1016/j.tins.2011.11.005
Edossa, A. K., Schroeders, U., Weinert, S., and Artelt, C. (2018). The development of emotional and behavioral self-regulation and their effects on academic achievement in childhood. Int. J. Behav. Dev. 42, 192–202. doi: 10.1177/0165025416687412
Espinoza Oyarce, D. A., Shaw, M. E., Alateeq, K., and Cherbuin, N. (2020). Volumetric brain differences in clinical depression in association with anxiety: a systematic review with meta-analysis. J. Psychiatry Neurosci. 45, 406–429. doi: 10.1503/jpn.190156
Eysenck, H. J., Eysenck, S. B. G., and Barrett, P. (1985). A revised version of the psychoticism scale. Personal. Individ. Differ. 6, 21–29. doi: 10.1016/0191-8869(85)90026-1
Fabrigar, L. R., and Wegener, D. T. (2012). Exploratory Factor Analysis. New York: Oxford University Press.
Franken, I. H., and Muris, P. (2005). Individual differences in reward sensitivity are related to food craving and relative body weight in healthy women. Appetite 45, 198–201. doi: 10.1016/j.appet.2005.04.004
Goldberg, D., Bridges, K., Duncan-Jones, P., and Grayson, D. (1988). Detecting anxiety and depression in general medical settings. BMJ 297, 897–899. doi: 10.1136/bmj.297.6653.897
Gray, J. A. (1970). The psychophysiological basis of introversion-extraversion. Behav. Res. Ther. 8, 249–266. doi: 10.1016/0005-7967(70)90069-0
Gray, J. A. (1982). Précis of the neuropsychology of anxiety: an enquiry into the functions of the septo-hippocampal system. Behav. Brain Sci. 5, 469–484. doi: 10.1017/S0140525X00013066
Gray, J.A., and Mcnaughton, N. (2000). The Neuropsychology of Anxiety: An Enquiry into the Functions of the Septo-Hippocampal System. New York: Oxford University Press.
Heubeck, B. G., Wilkinson, R. B., and Cologon, J. (1998). A second look at Carver and White's (1994) BIS/BAS scales. Personal. Individ. Differ. 25, 785–800. doi: 10.1016/S0191-8869(98)00124-X
Heym, N., Ferguson, E., and Lawrence, C. (2008). An evaluation of the relationship between Gray's revised RST and Eysenck's PEN: distinguishing BIS and FFFS in Carver and White's BIS/BAS scales. Personal. Individ. Differ. 45, 709–715. doi: 10.1016/j.paid.2008.07.013
Hu, L.-T., and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model. 6, 1–55. doi: 10.1080/10705519909540118
Jackson, C. J. (2009). Jackson-5 scales of revised reinforcement sensitivity theory (r-RST) and their application to dysfunctional real world outcomes. J. Res. Pers. 43, 556–569. doi: 10.1016/j.jrp.2009.02.007
Johnson, S. L., Turner, R. J., and Iwata, N. (2003). BIS/BAS levels and psychiatric disorder: an epidemiological study. J. Psychopathol. Behav. Assess. 25, 25–36. doi: 10.1023/A:1022247919288
Jöreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrika 36, 409–426. doi: 10.1007/BF02291366
Jorm, A. F., Christensen, H., Henderson, A. S., Jacomb, P. A., Korten, A. E., and Rodgers, B. (1999). Using the BIS/BAS scales to measure behavioural inhibition and behavioural activation: factor structure, validity and norms in a large community sample. Personal. Individ. Differ. 26, 49–58.
Kiely, K. M., and Butterworth, P. (2015). Validation of four measures of mental health against depression and generalized anxiety in a community based sample. Psychiatry Res. 225, 291–298. doi: 10.1016/j.psychres.2014.12.023
Kramer, S. L., and Rodriguez, B. F. (2018). A comparison of reinforcement sensitivity theory measures: unique associations with social interaction anxiety and social observation anxiety. Assessment 25, 627–639. doi: 10.1177/1073191116654003
Krupić, D., Corr, P. J., Ručević, S., Križanić, V., and Gračanin, A. (2016a). Five reinforcement sensitivity theory (RST) of personality questionnaires: comparison, validity and generalization. Personal. Individ. Differ. 97, 19–24. doi: 10.1016/j.paid.2016.03.012
Krupić, D., Gračanin, A., and Corr, P. J. (2016b). The evolution of the behavioural approach system (BAS): cooperative and competitive resource acquisition strategies. Personal. Individ. Differ. 94, 223–227. doi: 10.1016/j.paid.2016.01.044
Li, C. H. (2016). Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behav. Res. Methods 48, 936–949. doi: 10.3758/s13428-015-0619-7
Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika 58, 525–543. doi: 10.1007/BF02294825
Millsap, R. E., and Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivar. Behav. Res. 39, 479–515. doi: 10.1207/S15327906MBR3903_4
Mundfrom, D. J., Shaw, D. G., and Ke, T. L. (2005). Minimum sample size recommendations for conducting factor analyses. Int. J. Test. 5, 159–168. doi: 10.1207/s15327574ijt0502_4
Muris, P., Meesters, C., De Kanter, E., and Timmerman, P. E. (2005). Behavioural inhibition and behavioural activation system scales for children: relationships with eysenck’s personality traits and psychopathological symptoms. Personal. Individ. Differ. 38, 831–841. doi: 10.1016/j.paid.2004.06.007
Pagliaccio, D., Luking, K. R., Anokhin, A. P., Gotlib, I. H., Hayden, E. P., Olino, T. M., et al. (2016). Revising the BIS/BAS scale to study development: measurement invariance and normative effects of age and sex from childhood through adulthood. Psychol. Assess. 28, 429–442. doi: 10.1037/pas0000186
Pickering, A. D., and Gray, J. A. (1999). “The neuroscience of personality,” in Handbook of Personality: Theory and Research. eds. L. A. Pervin and O. P. John. 2nd ed (New York: The Guilford Press).
Poythress, N. G., Skeem, J. L., Weir, J., Lilienfeld, S. O., Douglas, K. S., Edens, J. F., et al. (2008). Psychometric properties of Carver and White's (1994) BIS/BAS scales in a large sample of offenders. Personal. Individ. Differ. 45, 732–737. doi: 10.1016/j.paid.2008.07.021
R Core Team (2019). “R: A Language and Environment for Statistical Computing“. 3.6.0 ed. (Vienna, Austria: R Foundation for Statistical Computing).
Reuter, M., Cooper, A. J., Smillie, L. D., Markett, S., and Montag, C. (2015). A new measure for the revised reinforcement sensitivity theory: psychometric criteria and genetic validation. Front. Syst. Neurosci. 9:38. doi: 10.3389/fnsys.2015.00038
Revelle, W. (2018). “Procedures for Personality and Psychological Research.” 1.8.12 ed. Evanston, Illinois, USA: Northwestern University.
Rosseel, Y. (2012). Lavaan: an R package for structural equation modeling. J. Stat. Softw. 48, 1–36. doi: 10.18637/jss.v048.i02
Smederevac, S., Mitrović, D., Čolović, P., and Nikolašević, Ž. (2014). Validation of the measure of revised reinforcement sensitivity theory constructs. J. Individ. Differ. 35, 12–21. doi: 10.1027/1614-0001/a000121
Smillie, L. D., Jackson, C. J., and Dalgleish, L. I. (2006a). Conceptual distinctions among Carver and White's (1994) BAS scales: a reward-reactivity versus trait impulsivity perspective. Personal. Individ. Differ. 40, 1039–1050. doi: 10.1016/j.paid.2005.10.012
Smillie, L. D., Pickering, A. D., and Jackson, C. J. (2006b). The new reinforcement sensitivity theory: implications for personality measurement. Personal. Soc. Psychol. Rev. 10, 320–335. doi: 10.1207/s15327957pspr1004_3
Svetina, D., and Rutkowski, L. (2017). Multidimensional measurement invariance in an international context: fit measure performance with many groups. J. Cross-Cult. Psychol. 48, 991–1008. doi: 10.1177/0022022117717028
Svetina, D., Rutkowski, L., and Rutkowski, D. (2019). Multiple-group invariance with categorical outcomes using updated guidelines: an illustration using Mplus and the lavaan/semTools packages. Struct. Equ. Model. 27, 111–130. doi: 10.1080/10705511.2019.1602776
Tiwari, T., Singh, A. L., and Singh, I. L. (2009). The short-form revised Eysenck personality questionnaire: a hindi edition (EPQRS-H). Ind. Psychiatry J. 18, 27–31. doi: 10.4103/0972-6748.57854
Van Tol, M.-J., Van Der Wee, N. J., Van Den Heuvel, O. A., Nielen, M. M., Demenescu, L. R., Aleman, A., et al. (2010). Regional brain volume in depression and anxiety disorders. Arch. Gen. Psychiatry 67, 1002–1011. doi: 10.1001/archgenpsychiatry.2010.121
Wardell, J. D., O'connor, R. M., Read, J. P., and Colder, C. R. (2011). Behavioral approach system moderates the prospective association between the behavioral inhibition system and alcohol outcomes in college students. J. Stud. Alcohol Drugs 72, 1028–1036. doi: 10.15288/jsad.2011.72.1028
Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54, 1063–1070. doi: 10.1037/0022-3514.54.6.1063
Windsor, T. D., Pearson, E. L., and Butterworth, P. (2012). Age group differences and longitudinal changes in approach–avoidance sensitivity: findings from an 8-year longitudinal study. J. Res. Pers. 46, 646–654. doi: 10.1016/j.jrp.2012.07.002
World Health Organization (2017). “Depression and Other Common Mental Disorders: Global Health Estimates. ” Geneva: World Health Organization.
Wu, H., and Estabrook, R. (2016). Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika 81, 1014–1045. doi: 10.1007/s11336-016-9506-0
Keywords: reinforcement sensitivity theory, factor analysis, measurement invariance, longitudinal, neuropsychology
Citation: Espinoza Oyarce DA, Burns R, Butterworth P and Cherbuin N (2021) Bridging Classical and Revised Reinforcement Sensitivity Theory Research: A Longitudinal Analysis of a Large Population Study. Front. Psychol. 12:737117. doi: 10.3389/fpsyg.2021.737117
Edited by:
Fernando Barbosa, University of Porto, PortugalReviewed by:
Snezana Smederevac, University of Novi Sad, SerbiaVilfredo De Pascalis, Sapienza University of Rome, Italy
Copyright © 2021 Espinoza Oyarce, Burns, Butterworth and Cherbuin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Daniela A. Espinoza Oyarce, ZGFuaWVsYWFuZHJlYS5lc3Bpbm96YW95YXJjZUBhbnUuZWR1LmF1