Event Abstract

Back to Event

Bidirectional synaptic scaling is necessary for implementing reinforcement learning via spike-timing-dependent-plasticity

Stefan Rach1*, Andreas Herzog1 and Christoph S. Herrmann1
  • 1 Carl-von-Ossietzky-Universität Oldenburg, Experimental Psychology Lab, Germany

The neuromodulator dopamine plays a key role in reinforcement learning processes. We adapted the spike timing-dependent plasticity (STDP) rule to implement reinforcement learning by assuming different STDP kernels for different dopamine levels (Gurney & Redgrave, 2008). At normal dopamine levels (equilibrium), the firing of a presynaptic neuron preceding a postsynaptic neuron results in long-term potentiation of synaptic transmission. If the dopamine level is increased following a reward, the magnitude of the long-term potentiation is also increased, whereas for decreased levels of dopamine following a punishment, the same temporal pattern of pre- and postsynaptic spikes leads to long-term depression.
However, if this mechanism is used in combination with a biologically plausible winner-take-all (WTA) architecture (Handrich et al., 2009) imperfect learning can occur. If the WTA mechanism always favors the same neuron on every trial and inhibits the other, no STDP mediated plastic changes can occur for the inhibited neuron, because it never fires.
In order to avoid this drawback, we explored a biologically plausible synaptic scaling mechanism which can prevent silencing of neurons by keeping a neuron’s firing rate within a dynamic range (Turrigiano, 2008). Using Izhikevich neurons (Izhikevich, 2004), we implemented a network with a WTA output layer that was able to learn via reinforcement learning.

Figure 1: (A) STDP template for low, normal, and high dopamine levels reflecting punishment, neutral, and reward conditions, respectively. (B) Synaptic scaling is achieved by scaling the synaptic weights during each time step by a factor that depends upon the firing rate. Below 10 Hz, synapses are scaled up and above 50 Hz they are scaled down.

Figure 1
Image

Acknowledgements

This work was supported by DFG (He3353/6-2) and BMBF (Bernstein Group Magdeburg).

References

• Gurney, K., Redgrave, P. (2008). A model of sensory reinforced corticostriatal plasticity in the anaethetised rat. Program No. 180.6/RR3. 2008 Neuroscience Meeting Planner. Washington, DC: Society for Neuroscience, 2008. Online.
• Handrich, S., Herzog, A., Wolf, A., & Herrmann, C. S. (2009). A Biologically PlausibleWinner-Takes-All Architecture. In D. Huang, K. Jo, H. Lee, H. Kang, & V. Bevilacqua, Emerging Intelligent Computing Technology and Applications. With Aspects of Artificial Intelligence, Lecture Notes in Computer Science (Vol. 5755, pp. 315-326). Berlin, Heidelberg: Springer
• Izhikevich, E. M. (2004). Which model to use for cortical spiking neurons? IEEE transactions on neural networks 15(5), 1063-1070.
• Turrigiano, G. G. (2008). The self-tuning neuron: synaptic scaling of excitatory synapses. Cell, 135(3), 422-435.

Keywords: computational neuroscience

Conference: Bernstein Conference on Computational Neuroscience, Berlin, Germany, 27 Sep - 1 Oct, 2010.

Presentation Type: Poster Abstract

Topic: Bernstein Conference on Computational Neuroscience

Citation: Rach S, Herzog A and Herrmann CS (2010). Bidirectional synaptic scaling is necessary for implementing reinforcement learning via spike-timing-dependent-plasticity. Front. Comput. Neurosci. Conference Abstract: Bernstein Conference on Computational Neuroscience. doi: 10.3389/conf.fncom.2010.51.00081

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 14 Sep 2010; Published Online: 23 Sep 2010.

* Correspondence: Dr. Stefan Rach, Carl-von-Ossietzky-Universität Oldenburg, Experimental Psychology Lab, Oldenburg, Germany, rach@leibniz-bips.de