Skip to main content

GENERAL COMMENTARY article

Front. Comput. Neurosci., 05 March 2015

A long journey into reproducible computational neuroscience

  • 1INRIA Bordeaux Sud-Ouest, Bordeaux, France
  • 2Institut des Maladies Neurodégénératives, Université de Bordeaux, UMR 5293, Bordeaux, France
  • 3Institut des Maladies Neurodégénératives, Centre National de la Recherche Scientifique, UMR 5293, Bordeaux, France
  • 4LaBRI, Université de Bordeaux, Institut Polytechnique de Bordeaux, Centre National de la Recherche Scientifique, UMR 5800, Talence, France
  • 5Centre de Neurophysique, Physiologie et Pathologies, Université Paris Descartes, Centre National de la Recherche Scientifique, UMR 8119, Paris, France

A commentary on
Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study

by Guthrie, M., Leblois, A., Garenne, A., and Boraud, T. (2013). J. Neurophysiol. 109, 3025–3040. doi: 10.1152/jn.00026.2013

Computational neuroscience is a powerful ally in our quest to understand the brain. Even the most simple model can shed light on the role of this or that structure and propose new hypothesis concerning the overall brain organization. However, any model in Science is doomed to be proved wrong or incomplete and replaced by a more accurate one. In the meantime, for such replacement to happen, we have first to make sure that models are actually reproducible such that they can be tested, evaluated, criticized and ultimately modified, replaced or even rejected. This is where the shoe pinches. If we cannot reproduce a model in the first place, we're doomed to re-invent the wheel again and again, preventing us from building an incremental computational knowledge of the brain.

We have been recently confronted to the problem when we tried to reproduce a model of the literature (Guthrie et al., 2013) concerning a computational model of the basal ganglia. This model was based on a previous modeling study by Leblois et al. (2006) where authors proposed an action selection mechanism based on a competition between the positive feedback, direct pathway through the striatum and the negative feedback, hyperdirect pathway through the subthalamic nucleus. Guthrie et al. (2013) further investigated how multiple level action selection could be performed, and the model has been extended in a manner consistent with known anatomy and electrophysiology of the basal ganglia in the monkey. The model is quite complex, but such is the basal ganglia. We asked authors for the sources of the model only to realize it has been implemented using 6000 lines of Delphi (Pascal language). We were unable to compile it (due to missing packages that we couldn't locate in any repository) and we thus decided to recode it from scratch. Unfortunately, the information provided in the article was not sufficient to allow for the direct reproduction of the model, mainly because there were factual errors in the manuscript and some information was ambiguous or missing. Ultimately, we were able to reach two of the original authors, T. Boraud, and A. Leblois (who are also authors of this commentary) in order to ask them about the details of the model. We joined efforts and proceeded with a complete rewrite, using the Python language (Perkel, 2015), a dedicated library (DANA), a versioning system (git), a public repository (github) and the IPython notebook, merely following the principles of reproducible computational science as proposed in Peng (2011), Sandve et al. (2013), Stodden et al. (2014). We also established the tabular description of the model as proposed in Nordlie et al. (2009). We claim this revamped model now allows any researcher in computational neuroscience to run it and to obtain the exact same results as the ones described in the original article (see https://hal.inria.fr/hal-01109483 for complete description and https://github.com/rougier/Neurosciences/tree/master/basal-ganglia/guthrie-et-al-2013 for source code). Furthermore, the new description, as well as the new figures, allow anyone to rewrite the model using a different language, tools or software.

However, the whole process took us approximately 3 months. This is hardly acceptable for the reproduction of a computational model that should be straightforward.

Unfortunately, this is not an isolated case. If computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing, and reproducing results, computational neuroscience still lags behind. In the best case, authors may provide the sources of their model as a compressed archive and feel confident their model is reproducible. But this is not exactly true. Buckheit and Donoho (1995) explained almost 20 years ago that, “an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data that produced the result.” The computational part in computational sciences implies the use of computers, operating systems, tools, frameworks, libraries, and data. This leads to such a large number of combinations (taking into account the version for each components) that the chances to have the exact same configuration as one of your colleague are nearly zero. This draws consequences in our respective computational approaches in order to make sure models and simulations can be actually and faithfully reproduced. We have to enforce the rules proposed in the literature and editors have to make sure this actually happens. From a broader perspective, this singular experience raises also some questions about the whole publication process. If articles remain the best media to publish a research and to introduce a model, why can't we have associated resources for the actual code just like we can have Supplementary Material as a separate document? For example, it is quite surprising that there is still no official code repository associated with journals. Even a dedicated public account on github (or any similar website) would really help on this matter.

But more importantly, given the quality of the new tools available today, it may be time to envisage new formats for communicating computational researches. For example, interactive documents could allow to replay a simulation and to modify parameters while reading the description of a model or a simulation. The IPython notebook is a serious candidate in that direction and could soon become a new way to exchange knowledge. It has been recently highlighted on Nature (Shen, 2014) and it is already widely used for teaching and writing books. Such new formats would definitely help authors, reviewers, readers and ultimately, Science.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Buckheit, J., and Donoho, D. L. (1995). “Wavelab and reproducible research,” in Wavelets and Statistics, Vol. 103, ed A. Antoniadis (New York, NY: Springer-Verlag), 55–81. doi: 10.1007/978-1-4612-2544-7_5

CrossRef Full Text

Guthrie, M., Leblois, A., Garenne, A., and Boraud, T. (2013). Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study. J. Neurophysiol. 109, 3025–3040. doi: 10.1152/jn.00026.2013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Leblois, A., Boraud, T., Meissner, W., Bergman, H., and Hansel, D. (2006). Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia. J. Neurosci. 26, 3567–3583. doi: 10.1523/JNEUROSCI.5050-05.2006

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Nordlie, E., Gewaltig, M., and Plesser, H. E. (2009). Towards reproducible descriptions of neuronal network models. PLoS Comput. Biol. 5:e1000456. doi: 10.1371/journal.pcbi.1000456

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Peng, R. D. (2011). Reproducible research in computational science. Science 334, 1226–1227. doi: 10.1126/science.1213847

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Perkel, J. M. (2015). Programming: pick-up python. Nature 518, 125–126. doi: 10.1038/518125a

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Sandve, G. K., Nekrutenko, A., Taylor, J., and Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS Comput. Biol. 9:e1003285. doi: 10.1371/journal.pcbi.1003285

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Shen, H. (2014). Interactive notebooks: sharing the code. Nature 515, 151–152. doi: 10.1038/515151a

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Stodden, V., Leisch, F., and Peng, R. D. (eds.). (2014). Implementing Reproducible Research. Chapman & Hall/CRC. Available online at: http://www.crcnetbase.com/doi/book/10.1201/b16868

Keywords: reproducible science, publication process, computational models, python language, version control, public repository, notebook

Citation: Topalidou M, Leblois A, Boraud T and Rougier NP (2015) A long journey into reproducible computational neuroscience. Front. Comput. Neurosci. 9:30. doi: 10.3389/fncom.2015.00030

Received: 30 January 2015; Accepted: 19 February 2015;
Published: 05 March 2015.

Edited by:

Florentin Wörgötter, University Goettingen, Germany

Reviewed by:

Florentin Wörgötter, University Goettingen, Germany
Timo Nachstedt, University of Goettingen, Germany

Copyright © 2015 Topalidou, Leblois, Boraud and Rougier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Nicolas P. Rougier, nicolas.rougier@inria.fr

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.