A correction has been applied to this article in:
Corrigendum: Functional MRI study of feedback-based reinforcement learning in depression
Almira M. Kustubayeva1,2
Erik B. Nelson2Michael L. Smith2,3
Jane B. Allendorfer2,4James C. Eliassen2,5*- 1Center for Cognitive Neuroscience, Department of Physiology, Biophysics, and Neuroscience, Al-Farabi Kazakh National University, Almaty, Kazakhstan
- 2Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States
- 3Department of Speech-Language-Hearing Sciences, University of Minnesota, Minneapolis, MN, United States
- 4Department of Neurology, University of Alabama at Birmingham, Birmingham, AL, United States
- 5Robert Bosch Automotive Steering LLC, Florence, KY, United States
Reinforcement learning depends upon the integrity of emotional circuitry to establish associations between environmental cues, decisions, and positive or negative outcomes in order to guide behavior through experience. The emotional dysregulation characteristic of major depressive disorder (MDD) may alter activity in frontal and limbic structures that are key to learning. Although reward and decision-making have been examined in MDD, the effects of depression on associative learning is less well studied. We investigated whether depressive symptoms would be related to abnormalities in learning-related brain activity as measured by functional magnetic resonance imaging (fMRI). Also, we explored whether melancholic and atypical features were associated with altered brain activity. We conducted MRI scans on a 4T Varian MRI system in 10 individuals with MDD and 10 healthy subjects. We examined event-related brain activation during feedback-based learning task using Analysis of Functional NeuroImages (AFNI) for image processing and statistical analysis. We observed that MDD patients exhibited reduced activation in visual cortex but increased activation in cingulate and insular regions compared to healthy participants. Also, in relation to features of depressive subtypes, we observed that levels of activation in striatal, thalamic, and precuneus regions were negatively correlated with atypical characteristics. These results suggest that the effects of MDD change the neural circuitry underlying associative learning, and these effects may depend upon subtype features of MDD.
Introduction
Major Depression Disorder is characterized by persistent, dysphoric emotions along with disturbances in motivated and psychomotor behavior, all of which interfere with activities of daily living. Depressive symptoms often accompany learning disabilities, but the evidence for the mechanisms of impaired learning in major depressive disorder (MDD) remains less certain (Hans, 1997; Bender et al., 1999). Learning yields behavioral adaptation to environmental changes by enabling individuals to associate positive and negative outcomes with prior behaviors. Learning disabilities may result from executive function deficits or emotion and reward processing abnormalities (Snyder, 2013; Knight and Baune, 2018; Wang et al., 2021). Empirical studies have reported impaired cognitive performance in MDD with attention and memory tasks related to hippocampal function (Weingartner et al., 1981; Dolan et al., 1992; Austin et al., 1999; Lin et al., 2014). Neuroimaging studies have identified brain dysfunction in patients with MDD during a variety of cognitive processes (Johnstone et al., 2007; Broyd et al., 2009; Smoski et al., 2009; Clark and Beck, 2010; Linden et al., 2012). Executive function deficits associated with altered prefrontal function and anterior cingulate cortex (ACC) activation have been described in several studies (Elliott et al., 1997, 1998; Austin et al., 2001). Impairments in emotion regulation related to prefrontal-subcortical abnormalities also contribute to depression vulnerability (Beauregard et al., 2006; Johnstone et al., 2007; Linden et al., 2012). A growing number of studies focusing on MDD symptom of anhedonia have demonstrated abnormalities in reward processing (Smoski et al., 2009, 2011; Zhang et al., 2013). The findings of altered reward processing are somewhat inconsistent. Hyposensitivity of striatal regions to reward feedback and reduced activation in the middle frontal gyrus and ACC have been reported in several studies (Schaefer et al., 2006; Forbes et al., 2009; Smoski et al., 2009). A meta-analysis of functional magnetic resonance imaging (fMRI) studies of reward processing in MDD supports the hypothesis of decreased subcortical and limbic activity in MDD patients (Zhang et al., 2013). However, other studies have reported hypersensitivity in the anterior insula to punishment and in the putamen to reward (Kumari et al., 2003; Remijnse et al., 2009), or no changes in reward areas during gain anticipation (Knutson et al., 2008). In these studies, the direction of effects to reward may depend on several factors including illness severity, medication history, and experimental task parameters. Impaired reward processing may underlie learning problems and behavioral adaptation (Chen et al., 2015). Reinforcement learning based on feedback is a more complex task which integrates sensory perception, reward processing, and motor action. Consequently, trial-and-error reinforcement learning may activate the different brain circuitry in comparison to simple reward processing. For instance, Gerraty et al. (2018) describe dynamical changes in brain networks and coupling between the striatum, visual, orbitofrontal, and ventromedial prefrontal cortex related to learning rate. Moreover, studies of reinforcement learning have been extended to computational models for outcome prediction (Daw et al., 2011) and its application to diagnostic and treatment methods (Brown et al., 2021; Heo et al., 2021). Therefore, it is important to understand the mechanism of reinforcement learning and how depressive symptoms such as emotion dysregulation and reward sensitivity affect the brain activation during learning. Also, positive and negative feedback itself may induce an emotional state corresponding to the valence of feedback (Kustubayeva et al., 2012). Previous learning research shows that healthy subjects exhibit prominent activation to the first trial feedback that diminishes on the next trial in a one-trial learning task (Eliassen et al., 2012). Feedback guides future behavior and the brain response to novel early feedback subsides dramatically as outcomes become predictable. It is important to learn how the integration of feedback and stimulus perception is changed in patients with MDD.
Our study evaluated the brain response to feedback-based visual-motor associative learning in healthy participants and MDD patients. We hypothesized that in comparison to healthy subjects MDD patients would show limbic-cortical dysregulation. We predicted that the presence of depression would lead to reduced activation during learning, suggesting less efficient information processing in MDD patients. Additionally, evidence from a number of studies suggests that depression is a heterogeneous disorder, with variations in phenomenology reflecting underlying neuropathophysiological differences (Fountoulakis et al., 2004; Baumeister and Parker, 2011; Foti et al., 2014; Day et al., 2015). This study recruited MDD patients who met criteria for a major depressive episode with either melancholic or atypical characteristics. Some studies show that atypical depression is distinguished from melancholia by increased right hemispheric processing and the right frontal perfusion (Fountoulakis et al., 2004). Because these subtypes differ with regard to neurovegetative symptoms and emotional reactivity, and sensitivity to reward, we also hypothesized that an altered response to learning would be associated with subtype characteristics. We present preliminary findings on these effects as well.
Materials and methods
Subjects
Twelve patients with MDD who met criteria for either atypical or melancholic subtypes of MDD (Mean Age = 40.5; SD = 7.92; 5 males, 7 females) and 10 healthy control subjects (Mean Age = 30.2; SD = 8.34; 5 males, 5 females) were recruited for this study via advertisement from outpatient and community populations. Inclusion criteria for both groups included age from 18 to 55, fluent in English and able to understand and provide written informed consent. MDD participants had to meet the following criteria: score on the Inventory of Depressive Symptomatology (IDS) scale higher than 20. Diagnosis of major depression was determined using the Structured Clinical Interview for DSM-IV (SCID), American Psychiatric Association [APA] (2000) and the presence of melancholic or atypical subtype was determined using the SCID supplements for these categories (Spitzer et al., 1990; First et al., 2002). Melancholic and atypical symptom score was based on sum of corresponding IDS items. Patients were unmedicated at the time of study participation and were excluded for use of any psychotropic medication in the past week, antidepressant medication, including herbal or natural substances purported to have antidepressant properties (e.g., St. John’s Wort, SAM-e, etc.) within the past 2 weeks, or the use of fluoxetine within the past 4 weeks. Two patients data were excluded due to the fMRI artifacts. To avoid the potential confounding effects of medication on blood oxygen level dependent (BOLD) signal changes, patients were included only if they had taken no antidepressant medications within 2 weeks of the scanning session. Also, participants were excluded if they had an unstable medical condition or and substance use disorder with the past 3 months.
Healthy subjects were included based on: (1) no history of DSM-IV Axis I disorder by history or by SCID interview; (2) no first-degree relative with a known history of a mood, anxiety, or substance use disorder. The study was approved by the Institutional Review Board (IRB) of the University of Cincinnati, and all subjects participated in the experiment after first providing written informed consent. Demographic, clinical and education information are summarized in Table 1. Education level was estimated by using the categories: some high school (0); graduated high school or Graduate Equivalency Degree (GED) (1); part college (2); completed 2 years of college (3); graduated 4 year college (4) and; graduate/professional school (5).
TABLE 1
Table 1. Demographic and symptom severity of the participants.
Behavioral testing
Neuroimaging paradigm
Following clinical and behavioral testing, patients participated in a functional neuroimaging session at the University of Cincinnati’s Center for Imaging Research (CIR). After arriving at the CIR each participant was given oral and written instructions and performed a brief practice of the task with pictures different from those used during scanning. Subjects were provided with video goggles to view the computer screen during scanning, and a button box held in the right hand was used to record participants’ responses to the task. MRI-compatible headphones and a microphone enabled communication with the patient during the scan.
Functional magnetic resonance imaging behavioral task
The associative learning task included both control and associative learning trials and was programmed in E-Prime1 to allow acquisition of behavioral performance (Figure 1; Eliassen et al., 2003, 2012). Two runs of the task were conducted during fMRI scanning. The control task consisted of 16 trials, and the learning task included 32 trials. During control trials the digit “1” or “2” was presented on the screen and participants pressed button 1 or 2 on the response box. During the learning trials, participants viewed easily named color pictures and learned to associate each picture with button 1 or 2 by trial-and-error using feedback (Rossion and Pourtois, 2004). A different set of four pictures was used for each run. Each picture was presented eight times pseudorandomized according to Latin Squares. A trial began with the presentation of a fixation cross (“X”). Following a brief variable delay a picture or digit appeared on screen, and subjects responded. After 2 s the screen went blank for a brief variable delay followed by feedback. Participants were instructed to make their responses during stimulus presentation. Positive feedback was indicated by a “+,” errors were indicated by “0,” and a lack of response resulted in a “?” Unlike our previous work with associative learning (Eliassen et al., 2012), the current task did not involve monetary rewards. For statistical analysis of the fMRI data we characterized learning by the number of correct responses to a picture. Learning or “experience” was divided into three levels, control, early learning (including the 1st trials, and the 2nd and 3rd correct trials) and late learning (all correct trials from 4th to 8th). Experience was then used as a factor in statistical analyses.
FIGURE 1
Figure 1. Schematic view of Associative Learning Task. The task included both control (N = 16) and learning trials (N = 32) in each run. The task was repeated twice during MRI scanning.
Functional magnetic resonance imaging experiment
Image acquisition
Participants were scanned in a 4T Varian INOVA MRI system. A high-resolution T1-weighted 3-dimensional anatomical image was acquired using a Modified Driven Equilibrium Fourier Transform (MDEFT) acquisition22 (TMD = 1.1 s, TR/TE = 13/6 ms, FOV = 256 × 192 × 192 mm, matrix 256 × 192 × 96 mm, voxel resolution 1 × 1 × 2 mm3, flip angle = 20°), zero-filled to 1 × 1 × 1 mm during reconstruction (Lee et al., 1995). For fMRI measurements we acquired two runs of T2*-weighted gradient-echo echoplanar images (EPI) consisting of 35 contiguous 5 mm-thick coronal slices covering the entire brain (TR/TE = 3,000/29 ms, FOV = 208 × 208 mm, matrix 64 × 64, flip angle = 75°, voxel resolution 3.25 × 3.25 × 5 mm) tuned to BOLD contrast and a multi-echo reference scan to correct geometric distortion and Nyquist ghost (Kwong et al., 1992; Ogawa et al., 1992; Schmithorst et al., 2001). Two 2-min runs were administered, and 146 volumes were acquired in each run. Two volumes at the beginning of each run were discarded to account for magnetization equilibrium effects.
MRI analysis
Functional images were reconstructed using in-house software written in Interactive Data Language (IDL)2 and included a 2D Hamming filter in the XY plane, which smoothed the resulting images. All further image processing including co-registration, motion correction, normalization, re-sampling, and event-related analysis were conducted in Analysis of Functional NeuroImages (AFNI) software (Cox, 1996; Cox and Hyde, 1997; Cox and Jesmanowicz, 1999). Fourier interpolation using a six-parameter rigid-body transformation was used for motion correction (Cox and Jesmanowicz, 1999). Each subject’s MDEFT image was normalized to Talairach space by alignment to the international consortium for brain mapping (ICBM) 452 brain template in AFNI.3 A 6 mm Gaussian smoothing was applied to the EPI data. EPI data sets were then normalized by adopting the MDEFT transform and resampled to 3 × 3 × 3 mm. Smoothing plus Hamming filtering resulted in 9 mm blurring in the XY (coronal) plane and 6 mm in the Z plane. The Monte Carlo simulations used to estimate statistical significance took this smoothing into account. Two subjects were excluded due to shim power supply hardware problems.
In order to estimate brain activation, a reference waveform was created for each subject by convolving specific task event times with a canonical hemodynamic response function (HRF) in AFNI. The canonical function incorporated a delay time of 2 s, a rise time of 4 s, and a fall time 4 s as we have used previously (Eliassen et al., 2012) and based on the timing of observed hemodynamic responses in our previous work (Eliassen et al., 2003). Signal drift was accounted for in each run by including polynomial regressors up to the third order (average, linear, quadratic, and cubic) and six motion correction parameters as regressors of no interest. Using AFNI’s 3dREMLfit, which accounts for serial auto correlations in the fMRI time series, we calculated fit coefficients to the polynomial regressors, the motion correction parameters and the reference waveforms representing the task events, stimulus, and feedback presentation for early learning, late learning, and control trials.
Statistical analyses
Behavioral analyses were conducted using a mixed effects analysis approach in Statistical Analysis Software (SAS Institute Inc., SAS 9.1.3 Help and Documentation, Cary, NC, United States: SAS Institute Inc., 2002–2004). Reaction time data was calculated by evaluating the change in response times across all levels of experience [Early: 1st trials (correct or incorrect), Correct 2nd, and Correct 3rd trials; Late: Correct 4th–8th trials; and Control Trials].
Analysis of Functional NeuroImages’ 3dMVM program, utilizing R for statistical computing, was used to examine the group fMRI data, accounting for repeated measures and subjects as a random factor. For fMRI the dependent measures were the fit coefficient maps representing control, early, and late learning for both stimulus and feedback events. Several analyses were conducted with the fMRI data. Two analyses compared depression patients to healthy comparison subjects: one examined stimulus processing and the other examined feedback. Factors in the analyses included group (patients vs. healthy), experience (early vs. late learning trials), and event types (stimulus vs. feedback). We included an interaction term for group by experience. We examined the correlation between atypical score and stimulus or feedback activation for learning trials. Because age was significantly lower in the healthy group (p < 0.01) we included age as a covariate in the statistical examination of behavioral and fMRI data. Functional Regions-of-interest (ROI) were obtained from the activation maps and average signal intensity was extracted for each individual for different task events. Significant thresholds were determined for all brain imaging comparisons according to the Monte Carlo simulation tools used in AFNI (Friston et al., 1993; Forman et al., 1995; Xiong et al., 1995). A significant cluster was defined as a corrected p-value of 0.01, using a voxel-level p-value ≤0.005 and a minimum cluster size of 37 voxels. Talairach daemon (Lancaster et al., 2000) was used in order to identify the locations of activation clusters as well as the Montreal Neurological Institutes (Tzourio-Mazoyer et al., 2002) automated anatomical labels of single-subject high resolution T1 volumes as implemented in AFNI.
Results
Behavior
Repeated measures analysis of variance showed a significant main effect of experience (F = 80.167, p < 0.0001), reflecting slower reaction time in early trials compared to late trials and control trials for all participants. There was no significant group difference (Figure 2A), no effect of age nor any significant interactions. There were no significant effects of group or age on error rate (Figure 2B).
FIGURE 2
Figure 2. (A) Reaction time as a function of experience, Early learning, Late learning, Control (Ctrl) in major depressive disorder (MDD) patients, and Healthy participants in the first and second runs. Learning is evident as RT decreases from Early to Late trials (Mean ± SD). (B) Total errors across groups (Mean ± SD) in MDD and Healthy groups in the first and second runs.
Imaging data
Brain activation during stimulus and feedback presentation
We examined brain activation differences between healthy and depressed patients and changes related to learning. Several brain regions exhibited the typical decline in activation with learning that we have observed previously. As with our previous research (Eliassen et al., 2003, 2012), these regions included bilateral medial superior frontal gyrus (BA6, 8), bilateral middle frontal gyrus (BA 9), and bilateral inferior frontal gyrus (anterior insula/frontal operculum; BA 44, 13) as well as parietal, temporal, and frontal polar regions.
Regions that showed a significant difference between groups included right fusiform gyrus (BA 38), right insula (BA 21), and right cingulate motor area (BA 6) (Figure 3) for the stimulus presentation. Interestingly activation for stimulus presentation was higher in occipital cortex in healthy participants, but in depressed patients was higher in insula and cingulate cortex. Post-hoc analyses of these three active clusters indicated no significant modulation by age.
FIGURE 3
Figure 3. Brain regions that showed significant group differences between major depressive disorder (MDD) and Healthy for stimulus presentation during learning (A), (B, C, D) Plots for ROI that displayed a significant difference between blood oxygen level dependent (BOLD) intensity in two groups. (B) Right Fusiform Gyrus; (C) Right insula; (D) Right Cingulate (motor area) cortex. Red color: MDD > Healthy; blue color: MDD < Healthy.
Atypical and melancholic scores and brain activation during learning
Analyses of subtype scores revealed several brain areas where activation levels during feedback presentation were correlated with atypical score (Figure 4). BOLD signal intensity on feedback during learning (early and late trials) negatively correlated with atypical score in right thalamus (including right hypothalamus and left ventral caudate) bilateral precuneus (BA31) right putamen, right inferior parietal, and left superior temporal gyrus. Activation in left cerebellum showed the opposite pattern with activation increasing with higher atypical score. Melancholic scores correlated positively with activation during late feedback in bilateral precuneus and right caudate (Table 2). There is no significant correlation for early feedback or for combined early and late feedback.
FIGURE 4
Figure 4. Brain regions that showed significant correlations for feedback presentation during learning and Atypical score in major depressive disorder (MDD) group (A), (B, C, D, E, F, G) Plots for ROI that displayed a significant correlation between blood oxygen level dependent (BOLD) intensity and Atypical score. (B)–Right Thalamus, (including right hypothalamus, and left ventral caudate); (C)–Bilateral Precuneus (BA31); (D)–Right Inferior Parietal Lobule; (E)–Left Cerebellum; (F)–Left Superior Temporal Gyrus; (G)–Right Putamen. Red color–positive correlation; blue color–negative correlation.
TABLE 2
Table 2. Regions with significant correlations between Melancholic score and feedback.
Discussion
The goal of the study was to identify differences between healthy individuals and MDD patients during reinforcement learning. Our results showed a similar learning effect in the behavioral and fMRI data: slower reaction time corresponded to increased BOLD intensity in the early learning trials compared to the late learning trials for both groups. The previous study (Eliassen et al., 2012) reported a similar learning effect in the behavioral and fMRI data in healthy participants. The current study observed the same effects in patients with MDD as well. The current study revealed significant differences in signals between healthy and depressed participants during learning. Correlations of the depression subtype scores to the BOLD signal are new findings of the brain activation changes during learning due to depression symptoms. Both groups showed decreased activation with learning, a pattern typical of healthy subjects according to our previous research (Eliassen et al., 2003, 2012). In comparison to healthy individuals, depressed patients showed elevated right insula and cingulate activation on stimulus presentation suggesting limbic hyperactivity during learning consistent with previous findings (Beauregard et al., 2006; Zhang et al., 2013). Elevated cingulate and insula activation may relate to general increased limbic activity in depression since it has been observed in the control trials as well. However, fMRI analyses revealed significant differences between groups in specific ROIs during the task performance and deviations became larger during the task performance in comparison to the already abnormal baseline activity, especially for decreased BOLD signal intensity in the right fusiform gyrus, and increased intensity in the right insula. Therefore, performing the task may provoke abnormal activity in learning circuits.
Greicius et al. (2007) reported abnormally increased functional connectivity in subgenual cingulate in depressed patients. Increased insula activation has been reported the neuronal basis of depression (Pandya et al., 2012). Additionally, anatomical studies showed that the gray matter volume of the insular cortex and the ACC correlated with depression symptoms (Mayberg, 1997). Electrophysiological studies on brain lateralization consider that the right hemisphere is hyperactive and the left hemisphere is hypoactive in patients with depression (Davidson, 1998; Allen and Reznik, 2015; Bruder et al., 2017). Our previous research confirmed left hypoactivity during the subsequent decision-making task in patients (Kustubayeva et al., 2020). FMRI study revealed changes in the right hemisphere in both directions depending on the brain structure.
We did not find significant reductions in activation on feedback presentation in MDD patients compared to healthy participants as shown in previous studies of reward hyposensitivity. Nevertheless, we must bear in mind that feedback during the current learning paradigm included negative feedback in early learning trials and positive feedback in the late learning that was predictable due to the easy task.
At the same time, in contrast to limbic regions, visual cortex activation was reduced in MDD patients compared to the control group, suggesting a decreased visual response to stimulus presentation (Schaefer et al., 2006; Knutson et al., 2008; Forbes et al., 2009; Smoski et al., 2009, 2011). This overall pattern suggests that limbic regions might activated to compensate for reduced visual cortical involvement, or that limbic overactivation disrupts visual cortical engagement. In either case, the limbic and cortical circuits are dysregulated in patients with MDD, which supports the characterization of MDD as a “multidimensional, system-level disorder” related to limbic-cortical circuit dysfunction (Mayberg, 1997, 2003).
A secondary aim of this study was to identify brain areas associated with atypical or melancholic features. The negative correlation with atypical score in thalamus, hypothalamus, and basal ganglia provides evidence of reward hyposensitivity in individuals with higher atypical scores. Higher melancholic scores were associated with higher brain activation in reward areas during late feedback (positive feedback only).
Interestingly, Foti et al. (2014) observed that event-related potential feedback negativity and fMRI ventral striatal responses to reward during guessing did not track melancholic or atypical features diagnosed by using the SCID. Authors found that feedback negativity (FN) was blunted in MDD subgroup with impaired mood reactivity. Therefore, further examination of the neural signature of depression subtypes and its diagnostic measurements is warranted to clarify these discrepant observations.
In conclusion, we observed differences in brain activation between MDD patients and healthy participants that provide evidence of limbic-cortical dysregulation during reinforcement learning that would not be predicted by reward hyposensitivity. On the other hand, brain activation in response to feedback correlated negatively with atypical features which suggests the presence of reward hyposensitivity in this subtype, and melancholic features revealed the opposite pattern. The presence of MDD is associated with dysregulation of neural circuits related to learning and subtype symptoms may alter the level of activity in some of these circuits.
Limitations
This study had limitations that bear comment. Due to the study requirement to include only unmedicated clinical samples only 10 patients were included to this study. We included age as covariate in the analysis because of age differences with the ten healthy volunteers. Our sample comprised of 10 MDD individuals with a range of subtype features, so we chose to focus on variation in activation across the continuum of atypical and melancholic features. Also, because our task was limited to two runs and learning occurs on the first trial regardless of outcome, we cannot distinguish activation between positive or negative feedback in early learning trials.
Data availability statement
Restrictions apply to the datasets. The datasets presented in this article are not readily available because our IRB protocol prohibits the public release of individual datasets. Requests to access the dataset should be directed to JE at amFtZXMuZWxpYXNzZW5AZ21haWwuY29t.
Ethics statement
The studies involving human participants were reviewed and approved by IRB, University of Cincinnati. The patients/participants provided their written informed consent to participate in this study.
Author contributions
AK was involved in behavioral, fMRI data analysis, and writing of the manuscript. MS was involved in data analysis and writing the method section. JA was involved in task design, data acquisition, and revision of the manuscript. JE and EN contributed to the research idea, in providing the research finding, designing and conducting the research, supervising the data acquisition and processing, and revision of the manuscript. All authors contributed to the article and approved the submitted version.
Funding
AK was supported by grant AP08856595 from the Ministry of Science and Higher Education of the Republic of Kazakhstan, EN was supported by a National Alliance for Research on Schizophrenia & Depression (NARSAD) Young Investigator Award and K23 MH067705, JA was supported by R01-DA022221-02S2 (PI: DelBello), and JE was supported by K01 DA020485.
Conflict of interest
Author JE was employed by Robert Bosch Automotive Steering LLC. Author JA serves as a consultant for LivaNova, Inc.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
References
Allen, J. J., and Reznik, S. J. (2015). Frontal EEG asymmetry as a promising marker of depression vulnerability: Summary and methodological considerations. Curr. Opin. Psychol. 4, 93–97. doi: 10.1016/j.copsyc.2014.12.017
American Psychiatric Association [APA] (2000). Diagnostic and statistical manual of mental disorders, 4th Edn. Washington, DC: American Psychiatric Association.
Austin, M. P., Mitchell, P., and Goodwin, G. M. (2001). Cognitive deficits in depression: Possible implications for functional neuropathology. Br. J. Psychiatry 178, 200–206. doi: 10.1192/bjp.178.3.200
Austin, M. P., Mitchell, P., Wilhelm, K., Parker, G., Hickie, I., Brodathy, H., et al. (1999). Melancholic depression: A pattern of frontal cognitive impairment. Psychol. Med. 29, 73–85. doi: 10.1017/S0033291798007788
Baumeister, H., and Parker, G. (2011). Meta-review of depressive subtyping models. J. Affect. Disord. 139, 126–140. doi: 10.1016/j.jad.2011.07.015
Beauregard, M., Paquette, V., and Levesque, J. (2006). Dysfunction in the neural circuitry of emotional self-regulation in major depressive disorder. Neuroreport 17, 843–846. doi: 10.1097/01.wnr.0000220132.32091.9f
Bender, W. N., Rosenkrans, C. B., and Crane, M. K. (1999). Stress, depression and suicide among students with learning disabilities: Assessing the risk. Learn. Disabil. Q. 22, 143–156. doi: 10.2307/1511272
Brown, V. M., Zhu, L., Solway, A., Wang, J. M., McCurry, K. L., King-Casas, B., et al. (2021). Reinforcement learning disruptions in individuals with depression and sensitivity to symptom change following cognitive behavioral therapy. JAMA Psychiatry 78, 1113–1122. doi: 10.1001/jamapsychiatry.2021.1844
Broyd, S. J., Demanuele, C., Debener, S., Helps, S. K., James, C. J., and Sonuga-Barke, E. J. (2009). Default-mode brain dysfunction in mental disorders: A systematic review. Neurosci. Biobehav. Rev. 33, 279–296. doi: 10.1016/j.neubiorev.2008.09.002
Bruder, G. E., Stewart, J. W., and McGrath, P. J. (2017). Right brain, left brain in depressive disorders: Clinical and theoretical implications of behavioral, electrophysiological and neuroimaging findings. Neurosci. Biobehav. Rev. 78, 178–191. doi: 10.1016/j.neubiorev.2017.04.021
Chen, C., Takahashi, T., Nakagawa, S., Inoue, T., and Kusumi, I. (2015). Reinforcement learning in depression: A review of computational research. Neurosci. Biobehav. Rev. 12, 247–267. doi: 10.1016/j.neubiorev.2015.05.005
Clark, D. A., and Beck, A. T. (2010). Cognitive theory and therapy of anxiety and depression: Convergence with neurobiological findings. Trends Cogn. Sci. 14, 418–424. doi: 10.1016/j.tics.2010.06.007
Cox, R. W. (1996). AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29, 162–173. doi: 10.1006/cbmr.1996.0014
Cox, R. W., and Hyde, J. S. (1997). Software tools for analysis and visualization of fMRI data. NMR Biomed. 10, 171–178. doi: 10.1002/(SICI)1099-1492(199706/08)10:4/5<171::AID-NBM453>3.0.CO;2-L
Cox, R. W., and Jesmanowicz, A. (1999). Real-time 3D image registration for functional MRI. Magn. Res. Med. 42, 1014–1018. doi: 10.1002/(SICI)1522-2594(199912)42:6<1014::AID-MRM4>3.0.CO;2-F
Davidson, R. J. (1998). Anterior electrophysiological asymmetries, emotion, and depression: Conceptual and methodological conundrums. Psychophysiology 35, 607–614. doi: 10.1017/S0048577298000134
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215. doi: 10.1016/j.neuron.2011.02.027
Day, C. V., Gatt, J. M., Etkin, A., DeBattista, C., Schatzberg, A. F., and Williams, L. M. (2015). Cognitive and emotional biomarkers of melancholic depression: An iSPOT-D report. J. Affect. Disord. 176, 141–150. doi: 10.1016/j.jad.2015.01.061
Dolan, R. J., Bench, C. J., Friston, K. J., Brown, R., Scott, L., and Frackowiak, R. S. J. (1992). Regional cerebral blood flow abnormalities in depressed patients with cognitive impairment. J. Neurol. Neurosurg. Psychiatry 55, 768–773. doi: 10.1136/jnnp.55.9.768
Eliassen, J. C., Lamy, M., Allendorfer, J. B., Boespflug, E., Bullard, D. P., Smith, M. S., et al. (2012). Selective role for striatal and prefrontal regions in processing first trial feedback during single-trial associative learning. Brain Res. 1458, 56–66. doi: 10.1016/j.brainres.2012.04.010
Eliassen, J. C., Souza, T., and Sanes, J. N. (2003). Experience-dependent activation patterns in human brain during visual–motor associative learning. J. Neurosci. 23, 10540–10547. doi: 10.1523/JNEUROSCI.23-33-10540.2003
Elliott, R., Baker, S. C., Rogers, R. D., O’Leary, D. A., Paykel, E. S., Frith, C. D., et al. (1997). Prefrontal dysfunction in depressed patients performing a planning task: A study using positron emission tomography. Psychol. Med. 27, 931–939. doi: 10.1017/S0033291797005187
Elliott, R., Sahakian, B. J., Michael, A., Paykel, E. S., and Dolan, R. J. (1998). Abnormal neural response to feedback on planning and guessing tasks in patients with unipolar depression. Psychol. Med. 28, 559–571. doi: 10.1017/S0033291798006709
First, M., Spitzer, R., Gibbon, M., and Williams, J. (2002). Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P). New York, NY: Biometrics Research Department, New York State Psychiatric Institute.
Forbes, E. E., Hariri, A. R., Martin, S. L., Silk, J. S., Moyles, D. L., Fisher, P. M., et al. (2009). Altered striatal activation predicting real-world positive affect in adolescent major depressive disorder. Am. J. Psychiatry 166, 64–73. doi: 10.1176/appi.ajp.2008.07081336
Forman, S. D., Cohen, J. D., Fitzgerald, M., Eddy, W. F., Mintun, M. A., and Noll, D. C. (1995). Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): Use of a cluster-size threshold. Magn. Reson. Med. 33, 636–647. doi: 10.1002/mrm.1910330508
Foti, D., Carlson, J. M., Sauder, C. L., and Proudfit, G. H. (2014). Reward dysfunction in major depression: Multimodal neuroimaging evidence for refining the melancholic phenotype. Neuroimage 101, 50–58. doi: 10.1016/j.neuroimage.2014.06.058
Fountoulakis, K. N., Iacovides, A., Gerasimou, G., Fotiou, F., Ioannidou, C., Bascialla, F., et al. (2004). The relationship of regional cerebral blood flow with subtypes of major depression. Prog. Neuropsychopharmacol. Biol. Psychiatry 28, 537–546. doi: 10.1016/j.pnpbp.2004.01.006
Friston, K. J., Worsley, K. J., Frackowiak, R. S. J., Mazziotta, J. C., and Evans, A. C. (1993). Assessing the significance of focal activations using their spatial extent. Hum. Brain Mapp. 1, 210–220. doi: 10.1002/hbm.460010306
Gerraty, R. T., Davidow, J. Y., Foerde, K., Galvan, A., Bassett, D. S., and Shohamy, D. (2018). Dynamic flexibility in striatal-cortical circuits supports reinforcement learning. J. Neurosci. 38, 2442–2453. doi: 10.1523/JNEUROSCI.2084-17.2018
Greicius, M. D., Flores, B. H., Menon, V., Glover, G. H., Solvason, H. B., Kenna, H., et al. (2007). Resting-state functional connectivity in major depression: Abnormally increased contributions from subgenual cingulated cortex and thalamus. Biol. Psychiatry 62, 429–437. doi: 10.1016/j.biopsych.2006.09.020
Hans, O. F. V. (1997). A preliminary profile of neuropsychological deficits associated with major depression. J. Clin. Exp. Neuropsychol. 19, 587–603. doi: 10.1080/01688639708403745
Heo, S., Sung, Y., and Lee, S. W. (2021). Effects of subclinical depression on prefrontal-striatal model-based and model-free learning. PLoS Comput. Biol. 17:e1009003. doi: 10.1371/journal.pcbi.1009003
Johnstone, T., Van Reekum, C. M., Urry, H. L., Kalin, N. H., and Davidson, R. J. (2007). Failure to regulate: Counterproductive recruitment of top-down prefrontal-subcortical circuitry in major depression. J. Neurosci. 33, 8877–8884. doi: 10.1523/JNEUROSCI.2063-07.2007
Knight, M. J., and Baune, B. T. (2018). Cognitive dysfunction in major depressive disorder. Curr. Opin. Psychiatry 31, 26–31. doi: 10.1097/YCO.0000000000000378
Knutson, B., Bhanji, J. P., Cooney, R. E., Atlas, L. Y., and Gotlib, I. H. (2008). Neural responses to monetary incentives in major depression. Biol. Psychiatry 63, 686–692. doi: 10.1016/j.biopsych.2007.07.023
Kumari, V., Mitterschiffthaler, M. T., and Teasdale, J. D. (2003). Neural abnormalities during cognitive generation of affect in treatment resistant depression. Biol. Psychiatry 54, 777–791. doi: 10.1016/S0006-3223(02)01785-7
Kustubayeva, A., Kamzanova, A., Kudaibergenova, S., Pivkina, V., and Matthews, G. (2020). Major depression and brain asymmetry in a decision-making task with negative and positive feedback. Symmetry 12:2118. doi: 10.3390/sym12122118
Kustubayeva, A., Matthews, G., and Panganiban, A. R. (2012). Emotion and information search in tactical decision-making: Moderator effects of feedback. Motiv. Emot. 36, 529–543. doi: 10.1007/s11031-011-9270-5
Kwong, K. K., Belliveau, J. W., Chesler, D. A., Goldberg, I. E., Weisskoff, R. M., Poncelet, B. P., et al. (1992). Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc. Natl. Acad. Sci. U.S.A. 89, 5675–5679. doi: 10.1073/pnas.89.12.5675
Lancaster, J. L., Woldorff, M. G., Parsons, L. M., Liotti, M., Freitas, C. S., Rainey, L., et al. (2000). Automated Talairach atlas labels for functional brain mapping. Hum. Brain Mapp. 10, 120–131. doi: 10.1002/1097-0193(200007)10:3<120::AID-HBM30>3.0.CO;2-8
Lee, J. H., Garwood, M., Menon, R., Adriany, G., Andersen, P., Truwit, C. L., et al. (1995). High contrast and fast three-dimensional magnetic resonance imaging at high fields. Magn. Reson. Med. 34, 308–312. doi: 10.1002/mrm.1910340305
Lin, F., Suhr, J., Diebold, S., and Heffner, K. L. (2014). Associations between depressive symptoms and memory deficits vary as a function of insulin-like growth factor (IGF-1) levels in healthy older adults. Psychoneuroendocrinology 42, 118–123. doi: 10.1016/j.psyneuen.2014.01.006
Linden, D. E. J., Habes, I., Johnston, S. J., Linden, S., Tatineni, R., Subramanian, L., et al. (2012). Real-time self-regulation of emotion networks in patients with depression. PLoS One 7:e38115. doi: 10.1371/journal.pone.0038115
Mayberg, H. S. (1997). Limbic-cortical dysregulation: A proposed model of depression. J. Neuropsychiatry Clin. Neurosci. 9, 471–481. doi: 10.1176/jnp.9.3.471
Mayberg, H. S. (2003). Modulating dysfunctional limbic-cortical circuits in depression: Towards development of brain-based algorithms for diagnosis and optimised treatment. Br. Med. Bull. 65, 193–207. doi: 10.1093/bmb/65.1.193
Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Merkle, H., et al. (1992). Intrinsic signal changes accompanying sensory stimulation: Functional brain mapping with magnetic resonance imaging. Proc. Natl. Acad. Sci. U.S.A. 89, 5951–5955. doi: 10.1073/pnas.89.13.5951
Pandya, M., Altinay, M., Malone, D. A. Jr., and Anand, A. (2012). Where in the brain is depression? Curr. Psyhiatry Rep. 14, 634–642. doi: 10.1007/s11920-012-0322-7
Remijnse, P. L., Nielen, M. M., van Balkom, A. J., Hendriks, G. J., Hoogendijk, W. J., Uylings, H. B., et al. (2009). Differential frontal-striatal and paralimbic activity during reversal learning in major depressive disorder and obsessive-compulsive disorder. Psychol. Med. 39, 1503–1518. doi: 10.1017/S0033291708005072
Rossion, B., and Pourtois, G. (2004). Revisiting Snodgrass and Vanderwart’s object pictorial set: The role of surface detail in basic-level object recognition. Perception 33, 217–236. doi: 10.1068/p5117
Schaefer, H. S., Putnam, K. M., Benca, R. M., and Davidson, R. J. (2006). Event-related functional magnetic resonance imaging measures of neural activity to positive social stimuli in pre- and post-treatment depression. Biol. Psychiatry 60, 974–986. doi: 10.1016/j.biopsych.2006.03.024
Schmithorst, V. J., Dardzinski, B. J., and Holland, S. K. (2001). Simultaneous correction of ghost and geometric distortion artifacts in EPI using a multiecho reference scan. IEEE Trans. Med. Imaging 20, 535–539. doi: 10.1109/42.929619
Smoski, M. J., Felder, J., Bizzell, J., Green, S. R., Ernst, M., Lynch, T. R., et al. (2009). fMRI of alterations in reward selection, anticipation, and feedback in major depressive disorder. J. Affect. Disord. 118, 69–78. doi: 10.1016/j.jad.2009.01.034
Smoski, M. J., Rittenberg, A., and Dichter, G. S. (2011). Major depressive disorder is characterized by greater reward network activation to monetary than pleasant image rewards. Psychiatry Res. 194, 263–270. doi: 10.1016/j.pscychresns.2011.06.012
Snyder, H. R. (2013). Major depressive disorder is associated with broad impairments on neuropsychological measures of executive function: A meta-analysis and review. Psychol. Bull. 139, 81–132. doi: 10.1037/a0028727
Spitzer, R., Williams, J., and Gibbon, M. (1990). Structured clinical interview for DSM-IV personality disorders (SCID-II). New York, NY: Biometrics Research.
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., et al. (2002). Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15, 273–289. doi: 10.1006/nimg.2001.0978
Wang, Y., Zhang, A., Yang, C., Li, G., Sun, N., Liu, P., et al. (2021). Enhanced functional connectivity within executive function network in remitted or partially remitted MDD patients. Front. Psychiatry 28:538333. doi: 10.3389/fpsyt.2020.538333
Weingartner, H., Kaye, W., Smallberg, S. A., Ebert, M. H., Gillin, J. C., and Sitaram, N. (1981). Memory failures in idiopathic dementia. J. Abnorm. Psychol. 90:187. doi: 10.1037/0021-843X.90.3.187
Xiong, J., Gao, J. H., Lancaster, J. L., and Fox, P. T. (1995). Clustered pixels analysis for functional MRI activation studies of the human brain. Hum. Brain Mapp. 3, 287–301. doi: 10.1002/hbm.460030404
Keywords: associative learning, reinforcement, functional magnetic resonance imaging (fMRI), major depressive disorder, brain activity
Citation: Kustubayeva AM, Nelson EB, Smith ML, Allendorfer JB and Eliassen JC (2022) Functional MRI study of feedback-based reinforcement learning in depression. Front. Neuroinform. 16:1028121. doi: 10.3389/fninf.2022.1028121
Received: 25 August 2022; Accepted: 18 October 2022;
Published: 20 December 2022.
Edited by:
Dmitrii Kaplun, Saint Petersburg State Electrotechnical University, RussiaReviewed by:
Haisan Zhang, Second Affiliated Hospital of Xinxiang Medical University, ChinaIvan V. Brak, State Scientific Research Institute of Physiology and Basic Medicine, Russia
Copyright © 2022 Kustubayeva, Nelson, Smith, Allendorfer and Eliassen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: James C. Eliassen, amFtZXMuZWxpYXNzZW5AZ21haWwuY29t