AUTHOR=Zorzi Marco , Testolin Alberto , Stoianov Ivilin P. TITLE=Modeling language and cognition with deep unsupervised learning: a tutorial overview JOURNAL=Frontiers in Psychology VOLUME=4 YEAR=2013 URL=https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2013.00515 DOI=10.3389/fpsyg.2013.00515 ISSN=1664-1078 ABSTRACT=

Deep unsupervised learning in stochastic recurrent neural networks with many layers of hidden units is a recent breakthrough in neural computation research. These networks build a hierarchy of progressively more complex distributed representations of the sensory data by fitting a hierarchical generative model. In this article we discuss the theoretical foundations of this approach and we review key issues related to training, testing and analysis of deep networks for modeling language and cognitive processing. The classic letter and word perception problem of McClelland and Rumelhart (1981) is used as a tutorial example to illustrate how structured and abstract representations may emerge from deep generative learning. We argue that the focus on deep architectures and generative (rather than discriminative) learning represents a crucial step forward for the connectionist modeling enterprise, because it offers a more plausible model of cortical learning as well as a way to bridge the gap between emergentist connectionist models and structured Bayesian models of cognition.