- 1 Network Imaging Group, Max Planck Institute for Biological Cybernetics, Tuebingen, Germany
- 2 Neurobiology Research Unit, Okinawa Institute of Science and Technology, Okinawa, Japan
- 3 Mind/Brain Institute and Department of Neurosciences, Johns Hopkins University, Baltimore, MD, USA
Spike timing dependent plasticity (STDP) is a temporally specific extension of Hebbian associative plasticity that has tied together the timing of presynaptic inputs relative to the postsynaptic single spike. However, it is difficult to translate this mechanism to in vivo conditions where there is an abundance of presynaptic activity constantly impinging upon the dendritic tree as well as ongoing postsynaptic spiking activity that backpropagates along the dendrite. Theoretical studies have proposed that, in addition to this pre- and postsynaptic activity, a “third factor” would enable the association of specific inputs to specific outputs. Experimentally, the picture that is beginning to emerge, is that in addition to the precise timing of pre- and postsynaptic spikes, this third factor involves neuromodulators that have a distinctive influence on STDP rules. Specifically, neuromodulatory systems can influence STDP rules by acting via dopaminergic, noradrenergic, muscarinic, and nicotinic receptors. Neuromodulator actions can enable STDP induction or – by increasing or decreasing the threshold – can change the conditions for plasticity induction. Because some of the neuromodulators are also involved in reward, a link between STDP and reward-mediated learning is emerging. However, many outstanding questions concerning the relationship between neuromodulatory systems and STDP rules remain, that once solved, will help make the crucial link from timing-based synaptic plasticity rules to behaviorally based learning.
Introduction
The first groundbreaking in vitro STDP studies seemed to paint a very clear picture: The near-coincidence of presynaptic input and postsynaptic spiking enables neurons to enhance or decrease their synaptic weights depending on the exact timing of these two events (Magee and Johnston, 1997; Markram et al., 1997; Bi and Poo, 1998; Debanne et al., 1998). This finding was a giant step forward in our view of synaptic plasticity rules: It tied together the idea that both single spikes and their precise timing matter; with the implication that neurons have a means to associate arriving inputs with the outgoing spikes and adapt the synaptic weights accordingly. It was therefore very intuitive to postulate STDP as a more temporally specific extension of Hebbian associative learning and experience driven plasticity (for definition, see below; and for modeling approaches see: Gerstner et al., 1993, 1996; Abbott and Blum, 1996; Blum and Abbott, 1996; Mehta et al., 2000).
However, for this idea to be relevant for behavioral learning as first formally proposed by Hebb (1949), it must hold true in vivo. This is where the STDP concept faces two conundrums, the first based on ongoing spiking activity, and the second based on the timing of spikes in relation to behavioral outcome.
(1) Unlike in vitro, in vivo there are continuous barrages of ongoing presynaptic activity impinging on the dendritic tree that can generate postsynaptic spikes (Figures 1A,B) (Wilson and Groves, 1981; Cowan et al., 1994; Wilson and Kawaguchi, 1996; Stern et al., 1997). These spikes backpropagate throughout the neuron (Waters et al., 2003; Waters and Helmchen, 2004), potentially interacting with the vast number of dendritically located synapses as these synapses continue to receive barrages of excitatory inputs (Figure 1C2). Under these conditions, the implications of STDP rules on individual synapses would be that a synapse active just prior to a spike event will increase in efficacy, whereas a synapse that is active just after the spike, will decrease its efficacy. The question arises, whether the mere association of presynaptic input and postsynaptic spiking activity would be enough to alter synaptic efficacy, and whether individual synapses in turn continuously scale up and down as inputs and backpropagating spikes constantly interact? Moreover, do spontaneously occurring spikes (Figure 1C2) and stimulus-evoked spikes (Figure 1C3) equally change synaptic weights as they interact with presynaptic input (Figure 1D)? When a spike is fired, whether it is spontaneous or evoked, how are active synaptic inputs that are driven by a stimulus separated from those that are due to the ongoing activity? One possible solution to these selectivity problems was originally proposed in relation to reward mediated learning (Miller, 1981; Wickens, 1990). These theoretical studies proposed that, in addition to the associated pre- and postsynaptic activity, a “third factor” was available to the network that enabled both the temporal and the spatial selection of specific inputs. To apply this to the in vivo situation, raises several further questions: could a neuromodulator represent such a third factor for selecting specific active inputs to a neuron that is embedded in a continuously active network? Given that many repetitions of timed pre–post pairings are typically necessary for STDP induction, could a third factor modify the number of repetitions needed for plasticity induction? In addition to the third factor requirement, other possible solutions for plasticity induction in vivo have been proposed that are not covered in the present review (Gerstner et al., 1996; Kempter et al., 1999; Beggs, 2001; Seung, 2003; Xie and Seung, 2004).
Figure 1. Sub- and suprathreshold neuronal activity in vivo and putative consequences for STDP. (A) Whole-cell recording from a pyramidal neuron in primary sensory cortex in vivo. Membrane potential trace contains (1) upstates, generated by presynaptic input, with no APs (action potentials: subthreshold events; subthr), (2) upstates with spontaneous APs (spont), and (3) upstates with APs evoked by sensory stimulation (sensory stim, indicated by bar). Hyperpolarizing current steps (I) were applied to determine input resistance. (B) Examples of spontaneous and stimulus-evoked activity in vivo. Events marked 1–3 in A (gray boxes) are depicted here in higher magnification. APs are truncated. (C) Dendritic interactions of presynaptic inputs during both subthreshold upstates and suprathreshold upstates with a spontaneous or a stimulus-evoked backpropagating AP. Left: Biocytine-stained pyramidal neuron showing soma, dendritic and axonal arborization. Part of the dendrite is shown schematically in the three panels at the right: (C1) During subthreshold events, upstate related synaptic input (up) arrives at dendritic spines. (C2) Spontaneous backpropagating APs (bAP spont) putatively interact with upstate related synaptic input arriving at plasticity-relevant timings. (C3) During sensory stimulation, stimulus-evoked backpropagating APs (bAP stim) can putatively interact with upstate related or with stimulus-evoked synaptic input (stim). (D) Putative changes in synaptic strength based on the timing of the AP with respect to incoming synaptic input (for both, upstate-related input [red] and/or stimulus-evoked input [green]). The question arises, if spontaneous bAPs as well as stimulus-evoked bAPs induce plasticity, when they interact with upstate related inputs (D2 vs. D3). In addition, the question arises, if both, stimulus-evoked and upstate-related input – when timed to coincide with bAPs – induce changes in synaptic strength (D3). Alternatively, in addition to timing, factors may exist that enable the spatial and temporal selection of activated synapses for plasticity.
(2) If spike timing dependent synaptic plasticity rules are the basis for the modification of behavior, and neuromodulatory systems are critical for this process, then a second conundrum of temporal credit assignment is faced. Both the behavioral signals and the behavioral outcome must be taken into account temporally. This likely also includes the activation of subcortical modulatory nuclei that can mediate for example alerting or rewarding signals to target structures (Schultz, 2000). How does such behavioral activation of a modulatory center influence the interaction of near-coincident pre- and postsynaptic activity with spatial and temporal specificity? Near-coincident pre- and postsynaptic spiking activity and neuromodulators most likely act on different timescales, ranging from tens of milliseconds for pre- and postsynaptic spikes to seconds or longer for some neuromodulators (for review see: Schultz, 2007). How will these different temporal activation schemes work together during behavioral learning? This temporal credit assignment problem is not new in neuroscience, as for example in reward mediated learning the “distal-reward problem” has been recognized years ago: How can the reward relate to specific events that happened earlier in time than the reward (Hull, 1943; Blum and Abbott, 1996; Schultz, 1998, 2006; Sutton and Barto, 1998; Izhikevich, 2007; Vasilaki et al., 2009)? Specific subcortical “reward systems” have been implicated in such learning with the neuromodulator dopamine being the most characterized (for review see: Schultz, 2000, 2002).
Although the rules associated with STDP have started to be addressed in vivo (Meliza and Dan, 2006; Jacob et al., 2007), to date, all the data about the involvement of neuromodulators in STDP have come from in vitro studies. In vitro, the dopaminergic system, amongst a number of other neuromodulatory systems, has been found to influence timing-dependent plasticity (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Lin et al., 2008; Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009). The following sections will present the existing in vitro experimental evidence concerning how neuromodulators are involved in timing-based plasticity. This review will be restricted to neuromodulator-actions on timing-dependent plasticity in the mammalian central nervous system. Furthermore, we concentrate on long-range neuromodulatory systems (that are thought to become activated by distinct behavioral states in vivo), although locally acting systems and retrograde messengers undoubtedly play an important role in STDP. Such locally acting systems important for STDP are for example endocannabinoids (Sjostrom et al., 2003; Bender et al., 2006; Nevian and Sakmann, 2006; Tzounopoulos et al., 2007), metabotropic glutamate receptors (mGluRs; Egger et al., 1999), and brain-derived neurotrophic factor (Mu and Poo, 2006; Sakata et al., 2009; Sivakumaran et al., 2009).
Experimental Evidence for Involvement of Neuromodulators in STDP
Neuromodulators are involved in most forms of synaptic plasticity ranging from short-term plasticity (ms) (for review see: Lovinger, 2010) to long-term plasticity (hours) (Neuman and Harley, 1983; Frey et al., 1990; Huerta and Lisman, 1993; Thomas et al., 1996), to experience-dependent plasticity (Bear and Singer, 1986; Kilgard and Merzenich, 1998) as well as structural plasticity (Ingham et al., 1998; Day et al., 2006; Gerfen, 2006) (for definitions, see below). Although over the past few decades the role of neuromodulation in certain forms of synaptic plasticity that mainly used high frequency stimulation induction protocols has been well established, it is not clear how these results relate to STDP (for reviews see: Jay, 2003; Hasselmo, 2006; Sara, 2009; Wickens, 2009).
To identify common neuromodulatory rules across the existing STDP studies is potentially difficult as not all studies have used the same induction protocols. The induction protocols used range from pairing of single spikes with a single synaptic input (Lin et al., 2003; Couey et al., 2007; Pawlak and Kerr, 2008; Zhang et al., 2009), and spike bursts with a single synaptic input (Seol et al., 2007; Shen et al., 2008) to “theta” burst paradigms, in which multiple excitatory postsynaptic potentials (EPSPs) are interleaved with multiple spikes (Bissiere et al., 2003; Shen et al., 2008) (see also Table 1). The use of multiple spikes with multiple EPSPs during theta burst protocols, usually evoked at around 30–50 Hz, does not allow true timing to be investigated as the preceding spike is always temporally close to the following evoked EPSP (for more details see: Froemke et al., 2010) but this may, in some cell types, be closer to what occurs during behavior. Studies into such complex pairing protocols indicate that the complex EPSP–spike interactions affect downstream signaling cascades differently to seemingly more simple EPSP–single spike interactions (Wang et al., 2005; Froemke et al., 2006). Also, other alterations to STDP recording conditions, for example the presence of GABAergic transmission or pre–post repetition rate may change the STDP window (for further reading see: Wickens, 2009). This opens the possibility that neuromodulators can activate different second messenger pathways depending upon the STDP induction protocol and recording conditions that were used. The identification of common neuromodulatory rules is further complicated by the use of different tools to manipulate neuromodulatory systems amongst studies, for example application of receptor agonists, receptor antagonists, or of the neuromodulator itself, often with different application times.
Table 1. Comparison of studies investigating the effect of neuromodulators on STDP.
Therefore, while we attempt to summarize neuromodulatory actions during timing-based plasticity in the following paragraph, one should be aware that differences in STDP induction protocols as well as method of neuromodulatory manipulation might impede finding common principles of neuromodulatory actions across studies.
Permission to Change: Neuromodulators and STDP
Studies investigating the effect of neuromodulators on STDP have either used a fixed relative pre–post timing to induce timing-dependent long-term potentiation and depression (t-LTP or t-LTD) (Bissiere et al., 2003; Couey et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008), or used a whole range of pre–post stimulation timings to investigate the effects of neuromodulation on the STDP timing window shape (Lin et al., 2003; Seol et al., 2007; Zhang et al., 2009). As will be discussed in detail in the following section, the studies using a fixed pre–post timing have often identified neuromodulatory signaling as a requirement for STDP to occur.
Of the neuromodulators investigated, dopamine is the most widely studied and has been shown to influence timing-dependent plasticity across several brain regions. In amygdala, t-LTP was only induced by a protocol consisting of short bursts of afferent stimulation timed to action potential (AP) bursts, when either dopamine was applied or GABAergic inhibition was blocked (Bissiere et al., 2003). Here, dopamine acted by activating dopamine D2 receptors, thereby suppressing feedforward inhibition from local interneurons, which permitted t-LTP induction by burst-pairing. The effect of dopamine depended on intact GABAergic transmission, since no potentiation occurred when dopamine was applied during pairing when GABAergic transmission was blocked, suggesting that the pairing protocol triggers different processes depending on the absence or presence of synaptic inhibition (Bissiere et al., 2003). In dorsal striatum under GABAA block, timing-dependent LTP was induced when a single AP closely followed an EPSP, while timing-dependent LTD was induced when the order was reversed (Pawlak and Kerr, 2008). Here, blocking dopamine D1/D5 receptors prevented t-LTP as well as t-LTD (Figures 2A,B), while blocking dopamine D2 receptors altered the onset, but not the final peak change in plasticity.
Figure 2. Timing-dependent LTP and LTD are under the control of dopamine D1/D5 receptors in striatal principal neurons. Anatomy of neuromodulatory fibers and the respective receptors as exemplified for striatal dopamine. (A) t-LTP was induced under control conditions (black circles) with a STDP protocol, where the AP followed the EPSP by 10 ms (Δt = 10 ms). (B) t-LTD was induced under control conditions with a protocol, where the EPSP followed the AP by 30 ms (Δt = −30 ms). No plasticity was observed with these two protocols, when dopamine D1/D5 receptors were blocked (green circles). (A,B modified from: Pawlak and Kerr, 2008). (C) Excitatory (glutamatergic) synapses arising from the cortex (Cx) or the thalamus (Th) onto spines of a striatal principal neuron. Only some of these spines also receive innervation from nigrostriatal (SN) dopaminergic fibers. Dopamine receptors (D1 and D2 subgroups) are distributed across distinct pre- and postsynaptic sites. For simplicity, the dopaminergic receptors, which are located on several of the striatal interneuron classes, are omitted from this cartoon.
Hence, the activation of dopamine D1/D5 receptors allowed for two events, the presynaptic input and the postsynaptic spike that occurred on a timescale of a few tens of milliseconds, to induce a lasting change in synaptic efficiency. Because dopamine receptors were blocked throughout the experiment with specific antagonists, the issue still remains whether dopamine alone acts in the same way when applied during the induction period (see “Changes to the Shape of the STDP Window”). In the subpopulation of striatal principal neurons that do not express dopamine D1 receptors, other neuromodulatory receptor systems were required for STDP. Here, adenosine A2 receptors, which are coupled to the similar second messenger cascades as D1 receptors (Premont et al., 1977; Schwarzschild et al., 2006) had to be activated for t-LTP induction. Both, endocannabinoid CB1 receptor as well as dopamine D2 receptor activation was required for t-LTD induction (Shen et al., 2008). In contrast to the amygdala, in the striatum the effect of dopamine on STDP seems to operate independently of GABAergic transmission (Pawlak and Kerr, 2008; Shen et al., 2008).
Nicotine was shown to be involved in STDP by acting on nicotinic acetylcholine receptors (nAChRs) in prefrontal cortex (Couey et al., 2007). Here, nicotine application caused normally t-LTP-inducing pre–post pairings, consisting of EPSP and single APs to induce a small amount of t-LTD in layer 5 pyramidal neurons. As an underlying mechanism, nicotine was found to strongly increase inhibition of layer 5 pyramidal neurons. Accordingly, the blocking effect of nicotine on t-LTP was partly overcome when inhibition was also blocked or a stronger t-LTP-inducing stimulus, consisting of pairing EPSPs with AP bursts, was applied. Since calcium is thought to be a crucial second messenger in synaptic plasticity induced by using spike-timing or other plasticity inducing protocols (for reviews see: Artola and Singer, 1993; Malenka and Bear, 2004), and since pre–post timing protocols produce characteristic spatiotemporal calcium signals (Koester and Sakmann, 1998), Couey et al. (2007) also investigated dendritic calcium dynamics during AP–EPSP pairing. Under the influence of nicotine, calcium changes were reduced during a pairing protocol that normally induced t-LTP in control condition (single-AP pairings). In contrast, during a stronger t-LTP-inducing protocol (AP-burst pairings), changes in dendritic calcium were comparable between control groups and nicotine groups. Both, pre- and postsynaptic nAChRs, distributed across several classes of interneurons, were suggested as potential targets of nicotine when reversing prefrontal t-LTP into t-LTD (Couey et al., 2007).
Not only manipulations of nicotinergic signaling, but also manipulations of the balance between dopaminergic and adenosinergic signaling are capable of reversing the sign of plasticity (i.e., converting t-LTP into t-LTD or vice versa); when dopaminergic signaling via D2 receptors was blocked and adenosine signaling was “boosted” by application of adenosine A2 receptor agonists, t-LTP was reversed into t-LTD upon a pre–post timing protocol (Shen et al., 2008).
In summary, dopamine receptor activation is often a prerequisite for timing-dependent plasticity to occur, and nicotine’s action is to increase the threshold for t-LTP induction. While the investigation of neuromodulatory influences using a few selective EPSP–AP timing protocols has led to important insights into specifically dopamine’s and nicotine’s actions during STDP, a more complete picture emerges when neuromodulatory influences are studied across the entire STDP window.
Changes to the Shape of the STDP Window
In hippocampus, unlike in striatum and amygdala, dopamine receptor activation was not a critical requirement for STDP induction. Here, t-LTP was induced by a pre–post protocol in the presence of dopamine receptor blockers (Zhang et al., 2009). However, dopamine application resulted in a modest, albeit not significant, increase in the amount of t-LTP observed when single postsynaptic APs closely followed the presynaptic activation with a delay of 10 ms. The much more dramatic effect observed with dopamine application was a change in the shape of the STDP window, allowing for longer pre–post timing delays to increase synaptic efficiency (Zhang et al., 2009). This widening effect was attributed to dopamine D1/D5 receptor activation, and was estimated to expand the t-LTP window by at least 25 ms. Surprisingly, t-LTD, as normally induced by a post–pre protocol, was converted into t-LTP by dopamine (Figure 3A). A similar broadening effect on the t-LTP window during the pairing of single APs with presynaptic activation was reported for the neuromodulator noradrenaline in hippocampal CA1 neurons (Lin et al., 2003). While the activation of β-adrenergic receptors widened the t-LTP window by about 15 ms, the overall amount of plasticity induced by the close pre–post pairings was not affected. Post–pre pairings were not tested. An unexpected similarity between both, β-adrenergic and dopaminergic actions on the “widening” of the t-LTP window is that the effect was expressed slowly, meaning that synaptic efficiency was unchanged directly after the pairing protocol and gradually started to increase from around 15 min post pairing protocol. The implications of window widening are that the activation of dopamine or noradrenaline receptors reduces the threshold for t-LTP induction by allowing for a wider range of pre–post timings to induce synaptic potentiation. The next question is whether the number of spikes needed to induce plasticity is altered with neuromodulation?
Figure 3. (A) Dopamine changes the shape of the STDP window in hippocampal neurons. STDP window in control conditions (black circles) and when dopamine was present during the STDP induction protocol (green circles). On the “t-LTP side” of the window (positive pre–post timings), dopamine allowed for longer intervals between spike and synaptic activation to induce potentiation of synaptic strength. On the “t-LTD side” of the window (negative pre–post timings), dopamine enabled t-LTP induction with a protocol that induced t-LTD under control conditions. (B) Dopamine reduces the number of spike pairs required to induce t-LTP. In control conditions, about 60 repetitions of timed pre–post spike pairings were required to induce robust t-LTP. In presence of dopamine, already 5–10 such pairings were sufficient to induce significant t-LTP. (A,B modified from: Zhang et al., 2009).
Changes to the Number of Trials Required to Achieve Plasticity
In addition to having an effect on the shape of the STDP window, dopamine also affected the number of pre–post pairing episodes required to induce plasticity in hippocampal neurons (Zhang et al., 2009). Specifically, when dopamine D1/D5 receptors were activated, successful t-LTP induction required a strongly reduced number of timed pre–post pairings, namely instead of the typically required 60 pairing trials, less than 10 pairings were required (Figure 3B). Thus by decreasing the required number of spike pairings, dopamine decreases the threshold for t-LTP induction.
Cooperation between Neuromodulators
As suggested by the anatomy of converging neuromodulatory fibers as well as direct physiological evidence, one neuromodulator often does not act in isolation, but several neuromodulatory systems interact (for example, see Bear and Singer, 1986; Zhou et al., 2001). In visual cortex layer 2/3 pyramidal neurons, pairing stimulation of layer 4 afferents with AP bursts did not result in plasticity, neither for pre–post protocols, nor for post–pre protocols (Seol et al., 2007; but compare Feldman, 2000; Froemke et al., 2005). For plasticity to occur, neuromodulatory receptors had to be activated during the pre–post timing protocols. Specifically under stimulation with β-adrenergic agonists, pre–post pairings, with timings between −50 and +50 ms, always induced t-LTP. Conversely, activation of M1 muscarinic receptors always resulted in t-LTD within the same range of timings. Finally, the “normal” standard STDP window displaying bidirectional plasticity, with causal pre–post timings leading to t-LTP and anticausal post–pre timings leading to t-LTD, was achieved with the combined application of β-adrenergic and M1 muscarinic agonists (Figure 4).
Figure 4. Coapplication of β-adrenergic and M1 muscarinic agonists is required for “standard” bidirectional STDP in visual cortex. In the presence of a β-adrenergic agonist alone, close positive as well as negative pre–post timings induced t-LTP (green circles). When a M1 muscarinic agonist was present, close positive as well as negative pre–post timings induced t-LTD (red circles). Only the combined application of β-adrenergic and M1 muscarinic agonists resulted in the “standard” STDP window with close pre–post timings leading to t-LTP, and post–pre timings leading to t-LTD (black circles). (Modified from: Seol et al., 2007).
The activation of β-adrenergic and M1 muscarinic receptors resulted in temporary phosphorylation of distinct sites at AMPA receptors that have been suggested to be crucial for t-LTP and t-LTD, respectively. This led the authors to conclude that neuromodulators supply AMPA receptors with distinct “tags” that allow during certain pre–post spiking timings the induction of t-LTP and t-LTD, respectively. In summary, the activation of noradrenergic and muscarinic acetylcholine receptors (mAChRs) is required for STDP, and more specifically, only the concurrent activation of the two neuromodulatory systems is required to achieve a “standard” STDP window with a t-LTP and a t-LTD side.
Conclusion of this Section
The observed neuromodulatory actions so far can be divided into two categories (see also Table 1): In the first category, neuromodulator receptor activation is necessary for plasticity (Bissiere et al., 2003; Seol et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008), thereby representing in addition to the precise timing of pre- and postsynaptic activity, a third factor essentially required for plasticity induction. Notably, in one study, two neuromodulators acted in concert to enable bidirectional STDP (Seol et al., 2007).
In the second category, the neuromodulator changes the conditions for plasticity by either increasing (Couey et al., 2007) or decreasing the threshold for plasticity induction ( Lin et al., 2003; Zhang et al., 2009; but see also Bienenstock et al., 1982). An effect observed in studies from both categories is that specific manipulations of one or several neuromodulator systems, result in sign reversal of plasticity, meaning that a normally t-LTP-inducing stimulus induced t-LTD, or vice versa (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Shen et al., 2008; Zhang et al., 2009). Although the mechanisms underlying such sign reversal are not clear, activity patterns that “boost” backpropagating APs in remote dendrites have been shown to “switch” t-LTD to t-LTP (Sjostrom and Hausser, 2006). Since some neuromodulators can exert a short-term effect on dendritic excitability and backpropagation (for review see: Waters et al., 2005), neuromodulators could also modulate backpropagating APs during STDP protocols, although this might not occur with all neuromodulators (Gulledge and Stuart, 2003).
The effect of neuromodulators on dendritic excitability is not restricted to short-term effects, since for example, the combination of mACh receptor activation with weak dendritic spikes in a distinct dendritic compartment resulted in a long-lasting excitability increase restricted to the involved dendritic compartment (Losonczy et al., 2008). This excitability increase transformed the weak dendritic spikes into strong dendritic spikes. Strong dendritic spikes have been implicated in drastic trial reduction to induce plasticity (Remy and Spruston, 2007). Such dendritic spikes are usually evoked with rather focused massive or convergent activity. Alternatively, a neuromodulator could reduce the trial number required for plasticity induction by making a dendritic compartment receptive for strong dendritic spike initiation (Losonczy et al., 2008).
From a temporal point of view, neuromodulators have been found to influence STDP on at least three timescales: on the scale of tens of milliseconds, neuromodulators influenced the interaction of pre- and postsynaptic spikes to induce plasticity; on the scale of seconds, neuromodulators influenced the number of repetitions of pre–post activity needed to evoke plasticity; and on the order of minutes, neuromodulators influenced the time course of plasticity.
Does the Third Factor have a Timing Issue, Too?
The studies listed in Section “Experimental Evidence for Involvement of Neuromodulators in STDP” have either constantly manipulated neuromodulator receptors during the entire experimental period (Lin et al., 2003; Pawlak and Kerr, 2008; Shen et al., 2008) or only during the induction period (Bissiere et al., 2003; Couey et al., 2007; Seol et al., 2007; Zhang et al., 2009). An important question that is very difficult to address experimentally, is how the outcome, in terms of plasticity, depends on the exact point in time of activation of neuromodulator-receptors, relative to pre- and postsynaptic spiking?
Since neuromodulator release sites and the receptors for the neuromodulator are not necessarily located close together on either side of the synaptic cleft, as in the classical concept of a synapse, the time required for diffusion of the released molecules has to be taken into account (see also Figure 2C). The modulator molecules have first to be released, then “travel” and bind to the respective receptors and initiate some G-protein coupled signaling cascade, which is a very different scenario from fast glutamate transmission. The time course of neuromodulator action was traditionally thought to be slow (on the scale of minutes), but recent evidence suggests that the time course is on the order of a few seconds (for review see: Sarter et al., 2009). Despite this recent change in thinking the question still arises how do these different timescales of spikes (1–2 ms) and neuromodulation (seconds) fit together in a working mechanism?
Three possible scenarios can be devised of how such a mechanism could work. The first two are at the single neuron level involving an “eligibility trace”, and the third is at the network level and relies on reverberating activity.
In the first scenario, the coordinated pre–post activity occurs before the neuromodulator release, as would be the case during unexpected reward. Here, spike and synaptic activation could leave a time decaying eligibility trace (Wang et al., 2000; Sarkisov and Wang, 2008) that subsequent neuromodulator receptor activation may then interact with to modulate plasticity.
In the second scenario, the neuromodulatory receptors are activated before the coordinated pre–post activity occurs, as would be expected during the learning of attention-based tasks. Here, the signaling mechanisms activated by neuromodulatory receptors themselves may create a slowly decaying eligibility trace, with which the coordinated pre–post activity can then interact with to modulate plasticity.
Experimental evidence for either scenario or the underlying molecular mechanisms is mostly lacking. However, the two presented scenarios resemble problems faced in the field of metaplasticity (Abraham, 2008) in which the concept of an eligibility trace has also been proposed. During metaplasticity, synapses will more easily undergo plasticity after a “priming” stimulus has changed the state of specific molecular signaling cascades; this change may for example “kick” plasticity-relevant enzymes into a more receptive state or it may result in enhanced phosphorylation of intracellular or extrasynaptic AMPARs that allows them to be inserted into postsynaptic membranes when an appropriate stimulus arrives (Sun et al., 2005; Abraham, 2008). For STDP, evidence for a similar gating mechanism in accordance with scenario two (neuromodulator receptor activation occurs first, followed by near-coincident pre–post spiking) was found by Seol and colleagues (see Sections “Experimental Evidence for Involvement of Neuromodulators in STDP and “Cellular and Molecular Targets of Neuromodulators During Timing-Dependent Plasticity”). When M1 muscarinic and β-adrenergic agonists were applied and washed out, a subsequent episode of timed pre–post pairings still initiated t-LTP or t-LTD, respectively. In addition, because neuromodulators have been shown to have a direct effect on glutamatergic receptor (AMPA and NMDA) location within the synapse and activated current efficacy (Seol et al., 2007; for review see: Cepeda and Levine, 2006), this is a possible mechanism that could create an eligibility trace to interact with subsequent pre–post pairing.
A third scenario how the three factors may interact in vivo, may be that the respective pre–post-activity patterns “reverberate” in the local circuit for some time (Hebb, 1949), and that such a memory trace can be transduced into a lasting modification if a third-factor success signal is present during the reverberation (Histed et al., 2009). A problem with such a mechanism is that the reverberating activity should not produce overt action, however if different circuits are involved, it is difficult to connect the success signal to the activity that did cause overt action.
How can Neuromodulators Influence the Interaction of Pre- and Postsynaptic Spikes: Anatomical and Physiological Considerations
Increasing evidence for the critical involvement of neuromodulator systems in STDP raises the question of how the physical location of neuromodulator release sites relates to the pre- and postsynaptic complex, which is thought to be the locus of STDP induction. Typically, neuromodulatory centers are located quite distally from the brain regions they influence (see more in Section “Activation of Neuromodulatory Systems In Vivo”). In their distal target areas, generally only a subgroup of neuromodulatory fibers makes direct contact with dendritic spines that receive excitatory inputs (Freund et al., 1984; Groves et al., 1994; Smith et al., 1994), whereas other neuromodulatory fibers target dendritic shafts and somata or form varicosities that lack synaptic specializations (Seguela et al., 1989, 1990). Therefore the question arises if only this subset of directly targeted synapses is influenced by neuromodulators when pre- and postsynaptic spikes collide? This is unlikely, because the receptors for the respective neurotransmitters are widely distributed across pre- and postsynaptic sites of principal neurons and interneurons (Gerfen et al., 1990, 1995; Sesack et al., 1994; Bergson et al., 1995; Caille et al., 1996) (Figure 2C), and receptor density and location can change (Paspalas et al., 2006). A certain degree of spatial specificity of neuromodulator-actions is probably achieved during behavior, when phasic release events occur, which temporally increases the concentration of neuromodulator, locally (reviewed in: Arbuthnott and Wickens, 2007; Sarter et al., 2009, see below for definition of phasic release and also Section “Activation of Neuromodulatory Systems In Vivo”). Finally, both AMPA and NMDA receptors are located presynaptically on neuromodulatory release terminals, and the activation of these receptors by overspill from neighboring active glutamatergic synapses is thought to convey further spatial specificity to the neuromodulator signal (Roberts and Sharif, 1978; Desce et al., 1992, 1994; Jin and Fredholm, 1994). Because the actions of neuromodulators are through receptors, the specific receptor subtype involved in STDP has important implications for the interpretation of neuromodulatory actions at the single neuron level. Indeed, during STDP, neuromodulators acted through specific receptor subtypes (Bissiere et al., 2003; Lin et al., 2003; Seol et al., 2007; Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009). This is a complex issue as, for example for the neuromodulator dopamine, low concentrations are thought to activate dopamine D2-like receptors in their high-affinity state, whereas high concentrations are thought to activate dopamine D1-like receptors (Richfield et al., 1989). These two receptor-subgroups are differentially expressed across neuronal populations, and can activate opposing downstream target enzymes (Girault and Greengard, 2004). Hence, during phasic release, the heterogeneous structural arrangement of release sites, different receptors subtypes, and regulated degrading/reuptake mechanisms in combination with diffusional processes are likely to generate further spatiotemporal heterogeneity of the neuromodulator signal. In addition, there is strong evidence that local spines within a dendritic region are topographically organized functionally (Jia et al., 2010) and that activity-initiated signaling cascades within the postsynaptic spines and dendrites interact locally with other spines (Harvey and Svoboda, 2007). This implies that in addition to the spatial specificity of neuromodulator release, there is a postsynaptic organization that can potentially provide very spatially defined neuromodulator action without the need for individual fibers to innervate each and every postsynaptic spine. The implication for timing-dependent plasticity in vivo would be that the timing of neuromodulator release in relation to correlated pre- and postsynaptic activity can enable the spatiotemporal selection of specific synapses for plasticity.
Cellular and Molecular Targets of Neuromodulators During Timing-Dependent Plasticity
Only few studies so far have addressed the issue of the exact cellular and molecular targets of neuromodulators when they “gate” STDP. In general, their receptors are (often) coupled to G-proteins and hereby influence intracellular second messenger cascades; (for example dopamine D1 receptors are coupled directly to adenylyl cyclase (AC) and indirectly to protein kinase A (PKA) and protein phosphatase 1 (Hemmings et al., 1984),M1 muscarinic ACh receptors are coupled to phospholipase C (PLC), β-adrenergic receptors are coupled to PKA (exception: nicotinic ACh receptors are ligand-gated channels). As a result, many different voltage-gated and calcium-dependent ion channels are influenced, which can affect membrane potential, neuronal spiking and excitatory transmission as well as inhibitory transmission (for review, see: Hasselmo, 1995; Nicola et al., 2000; Magee and Johnston, 2005; Sara, 2009). By their ability to affect dendritic ion channels, neuromodulators are certainly empowered to influence how the backpropagating AP will interact with incoming synaptic input during spike-timing paradigms (fHoffman and Johnston, 1999; Zhou et al., 2005; Sjostrom and Hausser, 2006; or review see: Tsubokawa, 2000), although specific studies investigating neuromodulatory influences on such interactions are required (but see: Couey et al., 2007).
Particularly in older animals, a preventing effect of inhibition on STDP has been described (Meredith et al., 2003). The influence that some neuromodulators have on inhibitory tone is certainly a means to affect STDP rules (D2, A2, mGluR5; Bissiere et al., 2003; Schwarzschild et al., 2006; Couey et al., 2007). However, several studies describe an effect of neuromodulators on STDP while inhibition is blocked, indicating at least one alternative mode of action of neuromodulators (Pawlak and Kerr, 2008; Shen et al., 2008).
Such an alternative mode of action is putatively an influence on the postsynaptic anchoring of glutamate receptors. For example, dopamine D1/D5 receptors and β-adrenergic receptors can increase surface expression of AMPA receptors (Chao et al., 2002; Sun et al., 2005; Oh et al., 2006) promoting synaptic insertion (for reviews see: Derkach et al., 2007; Lee and Huganir, 2008). The trafficking of AMPA receptors in and out of the synapse depends on phosphorylation of AMPA receptors at distinct sites (Lee et al., 2000, 2003; Boehm et al., 2006; He et al., 2009). In agreement with this, acetylcholine (coupled to PLC via M1 muscarinic receptors) and noradrenaline (coupled to AC via β-adrenergic receptors) gate phosphorylation at AMPA receptor sites implicated in t-LTP and t-LTD (Seol et al., 2007). In addition, β-adrenergic receptors were recently found to be anchored postsynaptically, forming a signaling complex with PKA and AMPA receptors (Joiner et al., 2010). Also, for dopamine, a complex interaction between D1 receptors and NMDA receptor channels has been reported (Cepeda et al., 1992; O’Donnell and Grace, 1994; Levine et al., 1996; Gao et al., 2001; Cepeda and Levine, 2006). Given that both dopamine and NMDA receptor activation were required for STDP (Pawlak and Kerr, 2008; Shen et al., 2008; Zhang et al., 2009), an interaction of these two receptor systems during the correlated pre- and postsynaptic spiking is possible.
Finally, it is worth considering the possibility that neuromodulators might also alter the dynamic balance of the phosphatases and kinases that control the induction of t-LTP and t-LTD (for review see: Lisman and McIntyre, 2001). For example, it is well established that PKA can reduce the activation of the phosphatases subserving LTD (Blitzer et al., 1998). This could be a plausible mechanism to account for the observation made in some studies that receptors coupled to AC, like D1 dopaminergic and β-adrenergic receptors, not only promote t-LTP but prevent t-LTD (Seol et al., 2007; Lin et al., 2008; Zhang et al., 2009).
In summary, the neuromodulatory systems can potentially affect STDP through a variety of mechanisms like changing the recruitment of inhibition at the network level, or changing excitability, kinase/phosphatase balance and priming AMPA receptor trafficking at the subcellular level. An important question that remains largely open is how those multiple actions are integrated in the different behavioral states defined by the neuromodulatory systems.
Activation of Neuromodulatory Systems in vivo
If one attempts a synthesis of STDP and neuromodulation, the question arises at which points during behavior neuromodulatory nuclei become activated? Due to tonic background activity of these nuclei, their innervated areas experience a constant low tone of release resulting in neuromodulator concentrations in the low nanomolar range. Salient behavioral events serve to drastically increase and decrease the activity of the respective nuclei (see below). The exact spatiotemporal profile of neuromodulator concentrations achieved during behavior is mostly unknown. Perhaps the best studied neuromodulator in this respect is dopamine, and some information about dopamine’s in vivo concentration is available (see below), whereas for noradrenaline and acetylcholine the concentration reached during behavior is not well studied.
Dopamine
Dopaminergic fibers arise from the ventral tegmental area and the substantia nigra. Dopaminergic neurons are activated by primary-rewarding stimuli: Unexpected rewards, but also the attentional and rewarding aspects of novel stimuli cause midbrain dopaminergic neurons to increase their firing rate (Ljungberg et al., 1992; Mirenowicz and Schultz, 1996). As a certain task is being learned, dopamine neurons shift their firing temporally toward the stimulus that indicates reward is to follow (Schultz et al., 1993). Hereby, the success-predicting stimulus has become rewarding. After a certain task has been learned, the primary-rewarding stimulus does not activate dopaminergic signals anymore; a dopaminergic signal is only initiated when a reward is unexpected or better than predicted. If a predicted reward is omitted, dopaminergic cells respond by decreasing their firing (Hollerman and Schultz, 1998). Together, this is consistent with theories of reinforcement learning stating that reinforcers only contribute to learning when they are not entirely predictable (Sutton and Barto, 1981). Recently, it has been found that a subpopulation of dopamine neurons also fire in response to aversive stimuli or associated cues (Joshua et al., 2008; Matsumoto and Hikosaka, 2009) suggesting that dopamine can code for multiple external events (Redgrave and Gurney, 2006).
The timescale of the phasic increase in firing rate of dopaminergic neurons is 50–110 ms (latency) and <200 ms (duration) with dopamine concentrations at target structures remaining elevated (150–400 nM) for up to 400 ms (Chergui et al., 1994; Dugast et al., 1994; Schultz, 2002). It is less clear how pauses in dopamine cell firing would affect local concentration levels, since the time course of clearance is relatively slow. However subtle changes in the degree of synchrony of firing have significant effects (Joshua et al., 2009).
Noradrenaline
Noradrenaline neurons located in locus coeruleus seem to play a role in vigilance, since these neurons show low firing rates during drowsiness and slow-wave sleep, regular firing at quiet wakefulness, and burst-firing in response to arousing stimuli (Aston-Jones and Bloom, 1981). A large variety of arousing and attention-demanding stimuli cause a response in these noradrenergic neurons, this also includes primary-rewarding stimuli and aversive stimuli (Foote et al., 1980; Rasmussen et al., 1986; Sara and Segal, 1991; Aston-Jones et al., 1994). In detail, this response consists of a very brief increase in AP firing (15–70 ms latency, 2–3 APs) followed by a longer suppression of AP firing (300–700 ms duration) (Berridge and Waterhouse, 2003). The noradrenergic response disappears with repeated stimulus presentations, but reappears when the stimulus is followed by reinforcement (Sara and Segal, 1991). In general, the noradrenaline signal is thought to be involved in sensory processing, decision-making, working memory, and memory formation (Cahill and McGaugh, 1996; Robbins and Roberts, 2007).
Acetylcholine
Cholinergic neurons in the basal forebrain nuclei are activated during arousal and attention (Paxinos, 2004; Sarter et al., 2005), they respond to unfamiliar stimuli (Wilson and Rolls, 1990), but also to unpredicted and predicted rewards (Richardson and DeLong, 1986, 1990). Also in striatum, tonically active striatal interneurons (TANs), which are cholinergic (Wilson et al., 1990; Zhou et al., 2002) respond to primary rewards and reward-predicting stimuli with a pause and sometimes a subsequent increase in firing (Aosaki et al., 1995; Apicella et al., 1997; Sardo et al., 2000; Zhou et al., 2002). The firing of TANs mainly encodes outcome delivery and omission at termination of the behavioral trial episode (Joshua et al., 2008). Within the cortex, acetylcholine has been suggested to enhance the response to sensory stimuli, and on more broad terms, to be important for attention and working memory (Hasselmo and Giocomo, 2006).
Conclusion of This Section
This section shows that release of neuromodulators occurs in a wide range of behavioral situations. Hence, the combination of theoretical and experimental work suggests that neuromodulatory influence on STDP might be linked to an equally wide range of behavioral learning processes, namely fear-conditioning (Bissiere et al., 2003), rapid learning (Zhang et al., 2009), reward-based learning (Pawlak and Kerr, 2008; Zhang et al., 2009), cognitive performance (Couey et al., 2007), but also pathological states (Shen et al., 2008) (for modeling approaches see: Baras and Meir, 2007; Florian, 2007; Legenstein et al., 2008; Vasilaki et al., 2009; Fremaux et al., 2010; Potjans et al., 2010).
Future Directions
Our knowledge about STDP and its regulation by neuromodulators has substantially increased during the last years, although the overall number of published studies concerning this topic remains low. Current experimental evidence suggests that neuromodulators shape the interaction between presynaptic and postsynaptic spike activity across many brain areas, and the predominating effect of neuromodulators is to allow plasticity or to make plasticity induction easier. Although there is amassing data from many different brain regions, it needs to be clarified how universal this additional modulatory factor is in regulating STDP. In addition, many brain areas are targeted and influenced by not only one, but by several neuromodulators (Bear and Singer, 1986; Zhou et al., 2001; Wang et al., 2006; Sara, 2009), and accordingly, there is experimental evidence that several neuromodulatory systems can interact to influence STDP rules (Seol et al., 2007). How the interaction of multiple neuromodulator systems exactly occurs during STDP, and if this interaction is a universal principle across many brain areas, will be interesting targets for future studies. Perhaps the most important outstanding question regarding STDP and neuromodulation concerns the exact time, when neuromodulatory receptors need to be activated to exert an influence on synaptic efficacy during causal and anticausal pre- and postsynaptic spiking. Along these lines, it will be important to directly test whether neuromodulators are capable of spatial or temporal selection of specific synapses for plasticity (see Sections “How Can Neuromodulators Influence the Interaction of Pre- and Postsynaptic Spikes: Anatomical and Physiological Considerations” and “Conclusion”).
Finally, not only will both experimentalists and theorists need to translate the effect of neuromodulators on STDP rules from in vitro to in vivo conditions (see: Meliza and Dan, 2006; Jacob et al., 2007), but also to the behaving animal. In addition, since it is almost certain that specific memories are stored across neuronal populations (Penfield, 1958, 1959), it will be important to see how STDP rules relate to neuronal populations in the behaving animal (Sawinski et al., 2009).
Conclusion
Spike timing dependent plasticity rules have been developed mainly on the basis of in vitro experimental data and have provided a temporally specific extension to the activity-based synaptic plasticity rules first proposed by Hebb (1949). However, when one tries to apply these rules to in vivo conditions and to the behaving animal, two conundrums arise: First, in vivo, a large amount of pre- and postsynaptic activity constantly arrives at the individual synapses. This raises the possibility that in vivo, synapses are constantly adapting their synaptic efficacy as pre- and postsynaptic spikes collide, which would be energetically inefficient for the involved neurons. An alternative possibility is that a “third factor” using a neuromodulator signal may represent a selection criteria that potentially allows presynaptic activity and postsynaptic spiking to be associated, both spatially and temporally. Thus, neuromodulators might enable the neuronal networks to select certain inputs and to make them eligible for changes in efficacy. To this end a large amount of indirect in vitro experimental evidence from many brain regions as well as theoretical evidence is being amassed that this may be the case, but a direct measurement of the third factor rule in vivo has yet to be achieved (for experimental evidence, see Section “Experimental Evidence for Involvement of Neuromodulators in STDP”; for modeling approaches see: Baras and Meir, 2007; Florian, 2007; Izhikevich, 2007; Legenstein et al., 2008; Vasilaki et al., 2009; Fremaux et al., 2010; Potjans et al., 2010).
Second, if one attempts to transfer the concepts of STDP to in vivo conditions, the obvious next questions are (a) if STDP rules are actually used for behaviorally based learning, and (b) how neuromodulation might be involved in this process. Neuromodulation alone is certainly an important factor involved in behavioral learning, as demonstrated by decades of research. If neuromodulation was instrumental in shaping STDP rules during behaviorally based learning, it would require fast time scale events like pre- and postsynaptic spikes and the putatively “slowly acting” neuromodulators to interact. The existing in vitro studies are only starting to provide insights how this temporal interaction might work. And for the in vivo situation, this picture will be much more complex, as a variety of behavioral states will release different combinations of neuromodulators at different timings and at different concentrations, activating different target receptor subtypes.
Dopamine, to date, is the most investigated neuromodulator and represents an interesting case for neuromodulator-regulation of STDP, as it has both the effect of “broadening” the t-LTP window and of changing what would normally be t-LTD into t-LTP. With regards to reward-mediated learning, the implication of these experimental findings is that any spike occurring within a certain window either before or after the synaptic input will increase the synaptic efficacy, which implies that many different external events that occurred temporally around the rewarding event could be associated with the reward.
It is an open question how an animal succeeds in linking the specific neuronal activity involved in a behavior to the behavioral outcome. How neuromodulators released during different states such as attention, arousal and reward influence this linking process, is also unknown. To achieve a full understanding of the principles of how neuromodulation shapes STDP rules might represent a first step toward solving these important questions.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Verena Pawlak and Jason N. D. Kerr were supported by the Max Planck Society. Alfredo Kirkwood was supported by NEI-R01-EY12124.
Glossary
Neuromodulator: A substance that is released by a neuron and alters the function of other neurons – typically on a slower timescale than a neurotransmitter.
Experience-dependent plasticity: Changes in synaptic strength or structural plasticity that result from manipulations altering sensory experience (Hooks and Chen, 2007; Fox, 2009).
Structural plasticity: Formation or elimination of dendritic spines, axonal boutons, synaptic contacts. Also includes structural rearrangements on a larger scale like changes in axonal/ dendritic arbors (Lamprecht and LeDoux, 2004; Holtmaat and Svoboda, 2009).
Synaptic change/synaptic plasticity: A change in the strength of synaptic transmission, which can be measured in several ways, like for example a change in the postsynaptic potential or postsynaptic current. It can be expressed on the level of a single neuron or a population of neurons (Bliss and Collingridge, 1993; Luscher et al., 2000).
Phasic release: Increase in neurotransmitter release that is restricted temporally.
References
Abbott, L. F., and Blum, K. I. (1996). Functional significance of long-term potentiation for sequence learning and prediction. Cereb. Cortex 6, 406–416.
Abraham, W. C. (2008). Metaplasticity: tuning synapses and networks for plasticity. Nat. Rev. Neurosci. 9, 387.
Aosaki, T., Kimura, M., and Graybiel, A. M. (1995). Temporal and spatial characteristics of tonically active neurons of the primate’s striatum. J. Neurophysiol. 73, 1234–1252.
Apicella, P., Legallet, E., and Trouche, E. (1997). Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioral states. Exp. Brain Res. 116, 456–466.
Artola, A., and Singer, W. (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends Neurosci. 16, 480–487.
Aston-Jones, G., and Bloom, F. E. (1981). Activity of norepinephrine-containing locus coeruleus neurons in behaving rats anticipates fluctuations in the sleep-waking cycle. J. Neurosci. 1, 876–886.
Aston-Jones, G., Rajkowski, J., Kubiak, P., and Alexinsky, T. (1994). Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J. Neurosci. 14, 4467–4480.
Baras, D., and Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural. Comput. 19, 2245–2279.
Bear, M. F., and Singer, W. (1986). Modulation of visual cortical plasticity by acetylcholine and noradrenaline. Nature 320, 172–176.
Beggs, J. M. (2001). A statistical theory of long-term potentiation and depression. Neural. Comput. 13, 87–111.
Bender, V. A., Bender, K. J., Brasier, D. J., and Feldman, D. E. (2006). Two coincidence detectors for spike timing-dependent plasticity in somatosensory cortex. J. Neurosci. 26, 4166–4177.
Bergson, C., Mrzljak, L., Smiley, J. F., Pappy, M., Levenson, R., and Goldman-Rakic, P. S. (1995). Regional, cellular, and subcellular variations in the distribution of D1 and D5 dopamine receptors in primate brain. J. Neurosci. 15, 7821–7836.
Berridge, C. W., and Waterhouse, B. D. (2003). The locus coeruleus-noradrenergic system: modulation of behavioral state and state-dependent cognitive processes. Brain Res. Brain Res. Rev. 42, 33–84.
Bi, G. Q., and Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464–10472.
Bienenstock, E. L., Cooper, L. N., and Munro, P. W. (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2, 32–48.
Bissiere, S., Humeau, Y., and Luthi, A. (2003). Dopamine gates LTP induction in lateral amygdala by suppressing feedforward inhibition. Nat. Neurosci. 6, 587–592.
Bliss, T. V., and Collingridge, G. L. (1993). A synaptic model of memory: long-term potentiation in the hippocampus. Nature 361, 31–39.
Blitzer, R. D., Connor, J. H., Brown, G. P., Wong, T., Shenolikar, S., Iyengar, R., and Landau, E. M. (1998). Gating of CaMKII by cAMP-regulated protein phosphatase activity during LTP. Science 280, 1940–1942.
Blum, K. I., and Abbott, L. F. (1996). A model of spatial map formation in the hippocampus of the rat. Neural. Comput. 8, 85–93.
Boehm, J., Kang, M. G., Johnson, R. C., Esteban, J., Huganir, R. L., and Malinow, R. (2006). Synaptic incorporation of AMPA receptors during LTP is controlled by a PKC phosphorylation site on GluR1. Neuron 51, 213–225.
Cahill, L., and McGaugh, J. L. (1996). Modulation of memory storage. Curr. Opin. Neurobiol. 6, 237–242.
Caille, I., Dumartin, B., and Bloch, B. (1996). Ultrastructural localization of D1 dopamine receptor immunoreactivity in rat striatonigral neurons and its relation with dopaminergic innervation. Brain Res. 730, 17–31.
Cepeda, C., and Levine, M. S. (2006). Where do you think you are going? The NMDA-D1 receptor trap. Sci. STKE 2006, pe20.
Cepeda, C., Radisavljevic, Z., Peacock, W., Levine, M. S., and Buchwald, N. A. (1992). Differential modulation by dopamine of responses evoked by excitatory amino acids in human cortex. Synapse 11, 330–341.
Chao, S. Z., Ariano, M. A., Peterson, D. A., and Wolf, M. E. (2002). D1 dopamine receptor stimulation increases GluR1 surface expression in nucleus accumbens neurons. J. Neurochem. 83, 704–712.
Chergui, K., Suaud-Chagny, M. F., and Gonon, F. (1994). Nonlinear relationship between impulse flow, dopamine release and dopamine elimination in the rat brain in vivo. Neuroscience 62, 641–645.
Couey, J. J., Meredith, R. M., Spijker, S., Poorthuis, R. B., Smit, A. B., Brussaard, A. B., and Mansvelder, H. D. (2007). Distributed network actions by nicotine increase the threshold for spike-timing-dependent plasticity in prefrontal cortex. Neuron 54, 73–87.
Cowan, R. L., Sesack, S. R., Van Bockstaele, E. J., Branchereau, P., Chain, J., and Pickel, V. M. (1994). Analysis of synaptic inputs and targets of physiologically characterized neurons in rat frontal cortex: combined in vivo intracellular recording and immunolabeling. Synapse 17, 101–114.
Day, M., Wang, Z., Ding, J., An, X., Ingham, C. A., Shering, A. F., Wokosin, D., Ilijic, E., Sun, Z., Sampson, A. R., Mugnaini, E., Deutch, A. Y., Sesack, S. R., Arbuthnott, G. W., and Surmeier, D. J. (2006). Selective elimination of glutamatergic synapses on striatopallidal neurons in Parkinson disease models. Nat. Neurosci. 9, 251–259.
Debanne, D., Gahwiler, B. H., and Thompson, S. M. (1998). Long-term synaptic plasticity between pairs of individual CA3 pyramidal cells in rat hippocampal slice cultures. J. Physiol. 507(Pt 1), 237–247.
Derkach, V. A., Oh, M. C., Guire, E. S., and Soderling, T. R. (2007). Regulatory mechanisms of AMPA receptors in synaptic plasticity. Nat. Rev. Neurosci. 8, 101–113.
Desce, J. M., Godeheu, G., Galli, T., Artaud, F., Cheramy, A., and Glowinski, J. (1992). L-glutamate-evoked release of dopamine from synaptosomes of the rat striatum: involvement of AMPA and N-methyl-D-aspartate receptors. Neuroscience 47, 333–339.
Desce, J. M., Godeheu, G., Galli, T., Glowinski, J., and Cheramy, A. (1994). Opposite presynaptic regulations by glutamate through NMDA receptors of dopamine synthesis and release in rat striatal synaptosomes. Brain Res. 640, 205–214.
Dugast, C., Suaud-Chagny, M. F., and Gonon, F. (1994). Continuous in vivo monitoring of evoked dopamine release in the rat nucleus accumbens by amperometry. Neuroscience 62, 647–654.
Egger, V., Feldmeyer, D., and Sakmann, B. (1999). Coincidence detection and changes of synaptic efficacy in spiny stellate neurons in rat barrel cortex. Nat. Neurosci. 2, 1098–1105.
Feldman, D. E. (2000). Timing-based LTP and LTD at vertical inputs to layer II/III pyramidal cells in rat barrel cortex. Neuron 27, 45–56.
Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural. Comput. 19, 1468–1502.
Foote, S. L., Aston-Jones, G., and Bloom, F. E. (1980). Impulse activity of locus coeruleus neurons in awake rats and monkeys is a function of sensory stimulation and arousal. Proc. Natl. Acad. Sci. U.S.A. 77, 3033–3037.
Fox, K. (2009). Experience-dependent plasticity mechanisms for neural rehabilitation in somatosensory cortex. Philos. Trans. R. Soc. Lond., B, Biol. Sci. 364, 369–381.
Fremaux, N., Sprekeler, H., and Gerstner, W. (2010). Functional requirements for reward-modulated spike-timing dependent plasticity. J. Neurosci. 30, 13326–13337.
Freund, T. F., Powell, J. F., and Smith, A. D. (1984). Tyrosine hydroxylase-immunoreactive boutons in synaptic contact with identified striatonigral neurons, with particular reference to dendritic spines. Neuroscience 13, 1189–1215.
Frey, U., Schroeder, H., and Matthies, H. (1990). Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res. 522, 69–75.
Froemke, R. C., Debanne, D., and Bi, G.-Q. (2010). Temporal modulation of spike-timing-dependent plasticity. Front. Syn. Neurosci. 2:12. doi: 10.3389/fnsyn.2010.00019.
Froemke, R. C., Poo, M. M., and Dan, Y. (2005). Spike-timing-dependent synaptic plasticity depends on dendritic location. Nature 434, 221–225.
Froemke, R. C., Tsay, I. A., Raad, M., Long, J. D., and Dan, Y. (2006). Contribution of individual spikes in burst-induced long-term synaptic modification. J. Neurophysiol. 95, 1620–1629.
Gao, W. J., Krimer, L. S., and Goldman-Rakic, P. S. (2001). Presynaptic regulation of recurrent excitation by D1 receptors in prefrontal circuits. Proc. Natl. Acad. Sci. U.S.A. 98, 295–300.
Gerfen, C. R. (2006). Indirect-pathway neurons lose their spines in Parkinson disease. Nat. Neurosci. 9, 157–158.
Gerfen, C. R., Engber, T. M., Mahan, L. C., Susel, Z., Chase, T. N., Monsma, F. J. Jr., and Sibley, D. R. (1990). D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science 250, 1429–1432.
Gerfen, C. R., Keefe, K. A., and Gauda, E. B. (1995). D1 and D2 dopamine receptor function in the striatum: coactivation of D1- and D2-dopamine receptors on separate populations of neurons results in potentiated immediate early gene response in D1-containing neurons. J. Neurosci. 15, 8167–8176.
Gerstner, W., Kempter, R., van Hemmen, J. L., and Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature 383, 76–81.
Gerstner, W., Ritz, R., and van Hemmen, J. L. (1993). Why spikes? Hebbian learning and retrieval of time-resolved excitation patterns. Biol. Cybern. 69, 503–515.
Girault, J. A., and Greengard, P. (2004). The neurobiology of dopamine signaling. Arch. Neurol. 61, 641–644.
Groves, P. M., Linder, J. C., and Young, S. J. (1994). 5-hydroxydopamine-labeled dopaminergic axons: three-dimensional reconstructions of axons, synapses and postsynaptic targets in rat neostriatum. Neuroscience 58, 593–604.
Gulledge, A. T., and Stuart, G. J. (2003). Action potential initiation and propagation in layer 5 pyramidal neurons of the rat prefrontal cortex: absence of dopamine modulation. J. Neurosci. 23, 11363–11372.
Harvey, C. D., and Svoboda, K. (2007). Locally dynamic synaptic learning rules in pyramidal neuron dendrites. Nature 450, 1195–1200.
Hasselmo, M. E. (1995). Neuromodulation and cortical function: modeling the physiological basis of behavior. Behav. Brain Res. 67, 1–27.
Hasselmo, M. E. (2006). The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 16, 710–715.
Hasselmo, M. E., and Giocomo, L. M. (2006). Cholinergic modulation of cortical function. J. Mol. Neurosci. 30, 133–135.
He, K., Song, L., Cummings, L. W., Goldman, J., Huganir, R. L., and Lee, H. K. (2009). Stabilization of Ca2+-permeable AMPA receptors at perisynaptic sites by GluR1-S845 phosphorylation. Proc. Natl. Acad. Sci. U.S.A. 106, 20033–20038.
Hebb, D. O. (1949). The Organization of Behavior. A Neuropsychological Theory. New York: John Wiley & Sons, Inc.
Hemmings, H. C. Jr., Greengard, P., Tung, H. Y., and Cohen, P. (1984). DARPP-32, a dopamine-regulated neuronal phosphoprotein, is a potent inhibitor of protein phosphatase-1. Nature 310, 503–505.
Histed, M. H., Pasupathy, A., and Miller, E. K. (2009). Learning substrates in the primate prefrontal cortex and striatum: sustained activity related to successful actions. Neuron 63, 244–253.
Hoffman, D. A., and Johnston, D. (1999). Neuromodulation of dendritic action potentials. J. Neurophysiol. 81, 408–411.
Hollerman, J. R., and Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309.
Holtmaat, A., and Svoboda, K. (2009). Experience-dependent structural synaptic plasticity in the mammalian brain. Nat. Rev. Neurosci. 10, 647–658.
Hooks, B. M., and Chen, C. (2007). Critical periods in the visual system: changing views for a model of experience-dependent plasticity. Neuron 56, 312–326.
Huerta, P. T., and Lisman, J. E. (1993). Heightened synaptic plasticity of hippocampal CA1 neurons during a cholinergically induced rhythmic state. Nature 364, 723–725.
Ingham, C. A., Hood, S. H., Taggart, P., and Arbuthnott, G. W. (1998). Plasticity of synapses in the rat neostriatum after unilateral lesion of the nigrostriatal dopaminergic pathway. J. Neurosci. 18, 4732–4743.
Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443–2452.
Jacob, V., Brasier, D. J., Erchova, I., Feldman, D., and Shulz, D. E. (2007). Spike timing-dependent synaptic depression in the in vivo barrel cortex of the rat. J. Neurosci. 27, 1271–1284.
Jay, T. M. (2003). Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog. Neurobiol. 69, 375–390.
Jia, H., Rochefort, N. L., Chen, X., and Konnerth, A. (2010). Dendritic organization of sensory input to cortical neurons in vivo. Nature 464, 1307–1312.
Jin, S., and Fredholm, B. B. (1994). Role of NMDA, AMPA and kainate receptors in mediating glutamate- and 4-AP-induced dopamine and acetylcholine release from rat striatal slices. Neuropharmacology 33, 1039–1048.
Joiner, M. L., Lise, M. F., Yuen, E. Y., Kam, A. Y., Zhang, M., Hall, D. D., Malik, Z. A., Qian, H., Chen, Y., Ulrich, J. D., Burette, A. C., Weinberg, R. J., Law, P. Y., El-Husseini, A., Yan, Z., and Hell, J. W. (2010). Assembly of a beta2-adrenergic receptor – GluR1 signalling complex for localized cAMP signalling. EMBO J. 29, 482–495.
Joshua, M., Adler, A., Mitelman, R., Vaadia, E., and Bergman, H. (2008). Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J. Neurosci. 28, 11673–11684.
Joshua, M., Adler, A., Prut, Y., Vaadia, E., Wickens, J. R., and Bergman, H. (2009). Synchronization of midbrain dopaminergic neurons is enhanced by rewarding events. Neuron 62, 695–704.
Kempter, R., Gerstner, W., and van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E 59, 4498.
Kilgard, M. P., and Merzenich, M. M. (1998). Cortical map reorganization enabled by nucleus basalis activity. Science 279, 1714–1718.
Koester, H. J., and Sakmann, B. (1998). Calcium dynamics in single spines during coincident pre- and postsynaptic activity depend on relative timing of back-propagating action potentials and subthreshold excitatory postsynaptic potentials. Proc. Natl. Acad. Sci. U.S.A. 95, 9596–9601.
Lamprecht, R., and LeDoux, J. (2004). Structural plasticity and memory. Nat. Rev. Neurosci. 5, 45–54.
Lee, H. K., Barbarosie, M., Kameyama, K., Bear, M. F., and Huganir, R. L. (2000). Regulation of distinct AMPA receptor phosphorylation sites during bidirectional synaptic plasticity. Nature 405, 955–959.
Lee, H. K., and Huganir, R. L. (2008). AMPA Receptor Regulation and the Reversal of Synaptic Plasticity – LTP, LTD, Depotentiation, and Dedepression, Vol. 4. San Diego: Elsevier Press.
Lee, H. K., Takamiya, K., Han, J. S., Man, H., Kim, C. H., Rumbaugh, G., Yu, S., Ding, L., He, C., Petralia, R. S., Wenthold, R. J., Gallagher, M., and Huganir, R. L. (2003). Phosphorylation of the AMPA receptor GluR1 subunit is required for synaptic plasticity and retention of spatial memory. Cell 112, 631–643.
Legenstein, R., Pecevski, D., and Maass, W. (2008). A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol. 4: e1000180. doi: 10.1371/journal.pcbi.1000180.
Levine, M. S., Li, Z., Cepeda, C., Cromwell, H. C., and Altemus, K. L. (1996). Neuromodulatory actions of dopamine on synaptically-evoked neostriatal responses in slices. Synapse 24, 65–78.
Lin, Y. W., Min, M. Y., Chiu, T. H., and Yang, H. W. (2003). Enhancement of associative long-term potentiation by activation of beta-adrenergic receptors at CA1 synapses in rat hippocampal slices. J. Neurosci. 23, 4173–4181.
Lin, Y. W., Yang, H. W., Min, M. Y., and Chiu, T. H. (2008). Inhibition of associative long-term depression by activation of beta-adrenergic receptors in rat hippocampal CA1 synapses. J. Biomed. Sci. 15, 123–131.
Lisman, J. E., and McIntyre, C. C. (2001). Synaptic plasticity: a molecular memory switch. Curr. Biol. 11, R788–R791.
Ljungberg, T., Apicella, P., and Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145–163.
Losonczy, A., Makara, J. K., and Magee, J. C. (2008). Compartmentalized dendritic plasticity and input feature storage in neurons. Nature 452, 436–441.
Lovinger, D. M. (2010). Neurotransmitter roles in synaptic modulation, plasticity and learning in the dorsal striatum. Neuropharmacology 58, 951–961.
Luscher, C., Nicoll, R. A., Malenka, R. C., and Muller, D. (2000). Synaptic plasticity and dynamic modulation of the postsynaptic membrane. Nat. Neurosci. 3, 545–550.
Magee, J. C., and Johnston, D. (1997). A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science 275, 209–213.
Magee, J. C., and Johnston, D. (2005). Plasticity of dendritic function. Curr. Opin. Neurobiol. 15, 334–342.
Markram, H., Lubke, J., Frotscher, M., and Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275, 213–215.
Matsumoto, M., and Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837–841.
Mehta, M. R., Quirk, M. C., and Wilson, M. A. (2000). Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25, 707–715.
Meliza, C. D., and Dan, Y. (2006). Receptive-field modification in rat visual cortex induced by paired visual stimulation and single-cell spiking. Neuron 49, 183–189.
Meredith, R. M., Floyer-Lea, A. M., and Paulsen, O. (2003). Maturation of long-term potentiation induction rules in rodent hippocampus: role of GABAergic inhibition. J. Neurosci. 23, 11142–11146.
Mirenowicz, J., and Schultz, W. (1996). Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379, 449–451.
Mu, Y., and Poo, M. M. (2006). Spike timing-dependent LTP/LTD mediates visual experience-dependent plasticity in a developing retinotectal system. Neuron 50, 115–125.
Neuman, R. S., and Harley, C. W. (1983). Long-lasting potentiation of the dentate gyrus population spike by norepinephrine. Brain Res. 273, 162–165.
Nevian, T., and Sakmann, B. (2006). Spine Ca2+ signaling in spike-timing-dependent plasticity. J. Neurosci. 26, 11001–11013.
Nicola, S. M., Surmeier, J., and Malenka, R. C. (2000). Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu. Rev. Neurosci. 23, 185–215.
O’Donnell, P., and Grace, A. A. (1994). Tonic D2-mediated attenuation of cortical excitation in nucleus accumbens neurons recorded in vitro. Brain Res. 634, 105–112.
Oh, M. C., Derkach, V. A., Guire, E. S., and Soderling, T. R. (2006). Extrasynaptic membrane trafficking regulated by GluR1 serine 845 phosphorylation primes AMPA receptors for long-term potentiation. J. Biol. Chem. 281, 752–758.
Paspalas, C. D., Rakic, P., and Goldman-Rakic, P. S. (2006). Internalization of D2 dopamine receptors is clathrin-dependent and select to dendro-axonic appositions in primate prefrontal cortex. Eur. J. Neurosci. 24, 1395–1403.
Pawlak, V., and Kerr, J. N. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J. Neurosci. 28, 2435–2446.
Penfield, W. (1958). Some mechanisms of consciousness discovered during electrical stimulation of the brain. Proc. Natl. Acad. Sci. U.S.A. 44, 51–66.
Penfield, W. (1959). The interpretive cortex; the stream of consciousness in the human brain can be electrically reactivated. Science 129, 1719–1725.
Potjans, W., Morrison, A., and Diesmann, M. (2010). Enabling functional neural circuit simulations with distributed computing of neuromodulated plasticity. Front. Comput. Neurosci. 4:141. doi: 10.3389/fncom.2010.00141
Premont, J., Perez, M., and Bockaert, J. (1977). Adenosine-sensitive adenylate cyclase in rat striatal homogenates and its relationship to dopamine- and Ca2+-sensitive adenylate cyclases. Mol. Pharmacol. 13, 662–670.
Rasmussen, K., Morilak, D. A., and Jacobs, B. L. (1986). Single unit activity of locus coeruleus neurons in the freely moving cat. I. During naturalistic behaviors and in response to simple and complex stimuli. Brain Res. 371, 324–334.
Redgrave, P., and Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967–975.
Remy, S., and Spruston, N. (2007). Dendritic spikes induce single-burst long-term potentiation. Proc. Natl. Acad. Sci. U.S.A. 104, 17192–17197.
Richardson, R. T., and DeLong, M. R. (1986). Nucleus basalis of Meynert neuronal activity during a delayed response task in monkey. Brain Res. 399, 364–368.
Richardson, R. T., and DeLong, M. R. (1990). Context-dependent responses of primate nucleus basalis neurons in a go/no-go task. J. Neurosci. 10, 2528–2540.
Richfield, E. K., Penney, J. B., and Young, A. B. (1989). Anatomical and affinity state comparisons between dopamine D1 and D2 receptors in the rat central nervous system. Neuroscience 30, 767–777.
Robbins, T. W., and Roberts, A. C. (2007). Differential regulation of fronto-executive function by the monoamines and acetylcholine. Cereb. Cortex 17(Suppl. 1), i151–i160.
Roberts, P. J., and Sharif, N. A. (1978). Effects of l-glutamate and related amino acids upon the release of [3H]dopamine from rat striatal slices. Brain Res. 157, 391–395.
Sakata, K., Woo, N. H., Martinowich, K., Greene, J. S., Schloesser, R. J., Shen, L., and Lu, B. (2009). Critical role of promoter IV-driven BDNF transcription in GABAergic transmission and synaptic plasticity in the prefrontal cortex. Proc. Natl. Acad. Sci. U.S.A. 106, 5942–5947.
Sara, S. J. (2009). The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 10, 211–223.
Sara, S. J., and Segal, M. (1991). Plasticity of sensory responses of locus coeruleus neurons in the behaving rat: implications for cognition. Prog. Brain Res. 88, 571–585.
Sardo, P., Ravel, S., Legallet, E., and Apicella, P. (2000). Influence of the predicted time of stimuli eliciting movements on responses of tonically active neurons in the monkey striatum. Eur. J. Neurosci. 12, 1801–1816.
Sarkisov, D. V., and Wang, S. S. (2008). Order-dependent coincidence detection in cerebellar Purkinje neurons at the inositol trisphosphate receptor. J. Neurosci. 28, 133–142.
Sarter, M., Hasselmo, M. E., Bruno, J. P., and Givens, B. (2005). Unraveling the attentional functions of cortical cholinergic inputs: interactions between signal-driven and cognitive modulation of signal detection. Brain Res. Brain Res. Rev. 48, 98–111.
Sarter, M., Parikh, V., and Howe, W. M. (2009). Phasic acetylcholine release and the volume transmission hypothesis: time to move on. Nat. Rev. Neurosci. 10, 383–390.
Sawinski, J., Wallace, D. J., Greenberg, D. S., Grossmann, S., Denk, W., and Kerr, J. N. (2009). Visually evoked activity in cortical cells imaged in freely moving animals. Proc. Natl. Acad. Sci. U.S.A. 106, 19557–19562.
Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115.
Schultz, W. (2007). Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288.
Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913.
Schwarzschild, M. A., Agnati, L., Fuxe, K., Chen, J. F., and Morelli, M. (2006). Targeting adenosine A2A receptors in Parkinson’s disease. Trends Neurosci. 29, 647–654.
Seguela, P., Watkins, K. C., and Descarries, L. (1989). Ultrastructural relationships of serotonin axon terminals in the cerebral cortex of the adult rat. J. Comp. Neurol. 289, 129–142.
Seguela, P., Watkins, K. C., Geffard, M., and Descarries, L. (1990). Noradrenaline axon terminals in adult rat neocortex: an immunocytochemical analysis in serial thin sections. Neuroscience 35, 249–264.
Seol, G. H., Ziburkus, J., Huang, S., Song, L., Kim, I. T., Takamiya, K., Huganir, R. L., Lee, H. K., and Kirkwood, A. (2007). Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55, 919–929.
Sesack, S. R., Aoki, C., and Pickel, V. M. (1994). Ultrastructural localization of D2 receptor-like immunoreactivity in midbrain dopamine neurons and their striatal targets. J. Neurosci. 14, 88–106.
Seung, H. S. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40, 1063–1073.
Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848–851.
Sivakumaran, S., Mohajerani, M. H., and Cherubini, E. (2009). At immature mossy-fiber-CA3 synapses, correlated presynaptic and postsynaptic activity persistently enhances GABA release and network excitability via BDNF and cAMP-dependent PKA. J. Neurosci. 29, 2637–2647.
Sjostrom, P. J., and Hausser, M. (2006). A cooperative switch determines the sign of synaptic plasticity in distal dendrites of neocortical pyramidal neurons. Neuron 51, 227–238.
Sjostrom, P. J., Turrigiano, G. G., and Nelson, S. B. (2003). Neocortical LTD via coincident activation of presynaptic NMDA and cannabinoid receptors. Neuron 39, 641–654.
Smith, Y., Bennett, B. D., Bolam, J. P., Parent, A., and Sadikot, A. F. (1994). Synaptic relationships between dopaminergic afferents and cortical or thalamic input in the sensorimotor territory of the striatum in monkey. J. Comp. Neurol. 344, 1–19.
Stern, E. A., Kincaid, A. E., and Wilson, C. J. (1997). Spontaneous subthreshold membrane potential fluctuations and action potential variability of rat corticostriatal and striatal neurons in vivo. J. Neurophysiol. 77, 1697–1715.
Sun, X., Zhao, Y., and Wolf, M. E. (2005). Dopamine receptor stimulation modulates AMPA receptor synaptic insertion in prefrontal cortex neurons. J. Neurosci. 25, 7342–7351.
Sutton, R. S., and Barto, A. G. (1981). Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170.
Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
Thomas, M. J., Moody, T. D., Makhinson, M., and O’Dell, T. J. (1996). Activity-dependent beta-adrenergic modulation of low frequency stimulation induced LTP in the hippocampal CA1 region. Neuron 17, 475–482.
Tsubokawa, H. (2000). Control of Na+ spike backpropagation by intracellular signaling in the pyramidal neuron dendrites. Mol. Neurobiol. 22, 129–141.
Tzounopoulos, T., Rubio, M. E., Keen, J. E., and Trussell, L. O. (2007). Coactivation of pre- and postsynaptic signaling mechanisms determines cell-specific spike-timing-dependent plasticity. Neuron 54, 291–301.
Vasilaki, E., Fremaux, N., Urbanczik, R., Senn, W., and Gerstner, W. (2009). Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Comput. Biol. 5, e1000586. doi: 10.1371/journal.pcbi.1000586.
Wang, H. X., Gerkin, R. C., Nauen, D. W., and Bi, G. Q. (2005). Coactivation and timing-dependent integration of synaptic potentiation and depression. Nat. Neurosci. 8, 187–193.
Wang, S. S., Denk, W., and Hausser, M. (2000). Coincidence detection in single dendritic spines mediated by calcium release. Nat. Neurosci. 3, 1266–1273.
Wang, Z., Kai, L., Day, M., Ronesi, J., Yin, H. H., Ding, J., Tkatch, T., Lovinger, D. M., and Surmeier, D. J. (2006). Dopaminergic control of corticostriatal long-term synaptic depression in medium spiny neurons is mediated by cholinergic interneurons. Neuron 50, 443–452.
Waters, J., and Helmchen, F. (2004). Boosting of action potential backpropagation by neocortical network activity in vivo. J. Neurosci. 24, 11127–11136.
Waters, J., Larkum, M., Sakmann, B., and Helmchen, F. (2003). Supralinear Ca2+ influx into dendritic tufts of layer 2/3 neocortical pyramidal neurons in vitro and in vivo. J. Neurosci. 23, 8558–8567.
Waters, J., Schaefer, A., and Sakmann, B. (2005). Backpropagating action potentials in neurones: measurement, mechanisms and potential functions. Prog. Biophys. Mol. Biol. 87, 145–170.
Wickens, J. (1990). Striatal dopamine in motor activation and reward-mediated learning: steps towards a unifying model. J. Neural Transm. Gen. Sect. 80, 9–31.
Wilson, C. J., Chang, H. T., and Kitai, S. T. (1990). Firing patterns and synaptic potentials of identified giant aspiny interneurons in the rat neostriatum. J. Neurosci. 10, 508–519.
Wilson, C. J., and Groves, P. M. (1981). Spontaneous firing patterns of identified spiny neurons in the rat neostriatum. Brain Res. 220, 67–80.
Wilson, C. J., and Kawaguchi, Y. (1996). The origins of two-state spontaneous membrane potential fluctuations of neostriatal spiny neurons. J. Neurosci. 16, 2397–2410.
Wilson, F. A., and Rolls, E. T. (1990). Neuronal responses related to the novelty and familarity of visual stimuli in the substantia innominata, diagonal band of Broca and periventricular region of the primate basal forebrain. Exp. Brain Res. 80, 104–120.
Xie, X., and Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E. Stat. Nonlin. Soft Matter Phys. 69, 041909.
Zhang, J. C., Lau, P. M., and Bi, G. Q. (2009). Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc. Natl. Acad. Sci. U.S.A. 106, 13028–13033.
Zhou, F. M., Liang, Y., and Dani, J. A. (2001). Endogenous nicotinic cholinergic activity regulates dopamine release in the striatum. Nat. Neurosci. 4, 1224–1229.
Zhou, F. M., Wilson, C. J., and Dani, J. A. (2002). Cholinergic interneuron characteristics and nicotinic properties in the striatum. J. Neurobiol. 53, 590–605.
Keywords: reward, learning, dopamine, acetylcholine, noradrenaline, synaptic plasticity, calcium, behavior
Citation: Pawlak V, Wickens JR, Kirkwood A and Kerr JND (2010) Timing is not everything: neuromodulation opens the STDP gate. Front. Syn. Neurosci. 2:146. doi: 10.3389/fnsyn.2010.00146
Received: 15 April 2010;
Accepted: 27 September 2010;
Published online: 25 October 2010.
Edited by:
Wulfram Gerstner, Ecole Polytechnique Fédérale de Lausanne, SwitzerlandReviewed by:
Markus Diesmann, RIKEN Brain Science Institute, JapanWulfram Gerstner, Ecole Polytechnique Fédérale de Lausanne, Switzerland
Henning Sprekeler, Ecole Polytechnique Fédérale de Lausanne, Switzerland
Nicolas Fremaux, Ecole Polytechnique Fédérale de Lausanne, Switzerland
Botond Szatmáry, Brain Corporation, USA
Copyright: © 2010 Pawlak, Wickens, Kirkwood and Kerr. This is an open-access article subject to an exclusive license agreement between the authors and the Frontiers Research Foundation, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited.
*Correspondence: Verena Pawlak and Jason N. D. Kerr, Network Imaging Group, Max Planck Institute for Biological Cybernetics, Spemannstrasse 41, 72076 Tuebingen, Germany. e-mail: verena.pawlak@tuebingen.mpg.de; jason@tuebingen.mpg.de