Xiaxia Xu
Ileana L. Hanganu-Opatz
Malte Bieler- 1Developmental Neurophysiology, Center for Molecular Neurobiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
- 2Laboratory for Neural Computation, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway
The emergence of cross-modal learning capabilities requires the interaction of neural areas accounting for sensory and cognitive processing. Convergence of multiple sensory inputs is observed in low-level sensory cortices including primary somatosensory (S1), visual (V1), and auditory cortex (A1), as well as in high-level areas such as prefrontal cortex (PFC). Evidence shows that local neural activity and functional connectivity between sensory cortices participate in cross-modal processing. However, little is known about the functional interplay between neural areas underlying sensory and cognitive processing required for cross-modal learning capabilities across life. Here we review our current knowledge on the interdependence of low- and high-level cortices for the emergence of cross-modal processing in rodents. First, we summarize the mechanisms underlying the integration of multiple senses and how cross-modal processing in primary sensory cortices might be modified by top-down modulation of the PFC. Second, we examine the critical factors and developmental mechanisms that account for the interaction between neuronal networks involved in sensory and cognitive processing. Finally, we discuss the applicability and relevance of cross-modal processing for brain-inspired intelligent robotics. An in-depth understanding of the factors and mechanisms controlling cross-modal processing might inspire the refinement of robotic systems by better mimicking neural computations.
Sensory-Cognitive Interplay During Cross-Modal Processing
The brain permanently receives sensory information addressing multiple modalities. Its capability to process diverse sensory inputs is mandatory to create a coherent perception of the environment, and ultimately to guide adaptive behavior. The diverse sensory components of a stimulus are processed and conveyed in a discrete manner by modality-specific pathways (Figure 1A), where each modality provides unique information about the stimulus. Complementing stimulus information reduces stimulus uncertainty and enhances behavioral responses, thus leading to faster and more accurate decision-making (Stein et al., 1988; Gleiss and Kayser, 2012; Siemann et al., 2014; Hammond-Kenny et al., 2017; Meijer et al., 2018). The process of sensory convergence, where inputs of different senses are combined without being able to easily dismantle them into independent unimodal components, is termed as cross-modal integration (Keil and Senkowski, 2018; Nikbakht et al., 2018). In order to evoke a coherent cross-modal perception, neural areas accounting for sensory and cognitive processing need to optimally interact with each other. This appears to be a challenging computation given the multidimensionality of neural activity and the fact that neural areas specialized in processing one component of a stimulus are located at distant parts in the brain (Harris and Mrsic-Flogel, 2013; Runyan et al., 2017; Stringer et al., 2019). In addition, the neural interactions of systems accounting for sensory and cognitive processing are highly dynamic, emerging at early age and developing over time (Goodman and Shatz, 1993; Siegel et al., 2012; Parisi et al., 2019). Comparable sensory systems and the ease of measuring behavioral effects motivated the use of large mammalian species as prime models to study the mechanisms of cross-modal processing and their emergence during development (Stein et al., 1993; Wallace and Stein, 1997; Calvert and Thesen, 2004). Here we focus on the interdependence of primary sensory cortices (S1, V1, A1) and PFC in rodents, aiming to critically review our current understanding of the mechanisms that enable the communication between remote brain areas dedicated to sensory and cognitive processing during cross-modal perception. In addition, we will review how bottom-up and top-down mechanisms underlying cross-modal processing emerge during development. Despite possible differences of neuronal processing when compared to larger mammals such as cats or monkeys, the use of rodent models bears several advantages for the study of cross-modal processing. Recent developments in rodent behavior and genetics, viral methods, and genetically encoded Ca2+ indicators offer the possibility to study causal relations in the brain, monitor neuronal activity over time, and explore the relationship between neural network properties and behavior underlying cross-modal processing (Fenno et al., 2011; Chen et al., 2013). Relying on these state-of-the-art methods, our understanding of the cellular and network mechanisms underlying cross-modal processing as well as their development should be fostered. Detailed insights on the neural computations are critical for the development of autonomous agents and their optimal interaction with the environment under conditions of sensory uncertainty. Thus, by providing knowledge of neuronal computations underlying cross-modal integration, this review aims to uncover general principles of neuronal processing and to inspire multidisciplinary research in the field of robotics.
Figure 1. Bottom-up and top-down cross-modal processing. (A) Schematic drawing of a mouse receiving visual information (red arrow) about a behaviorally-irrelevant object (trees) and a behaviorally relevant object (approaching eagle) that is accompanied by tactile and auditory information (vibrations and sounds, green and blue arrows). (i) Schematic diagram showing how visual (ii), tactile (iii), and auditory (iv) information is transferred in the brain. (B) Schematic diagram of bottom-up sensory information flow from primary sensory cortices to PFC. The black arrows correspond to cross-modal processing from primary sensory cortices to PFC, whereas the gray arrows correspond to cross-modal processing within primary sensory cortices. (C) Schematic diagram of top-down prefrontal modulation of neuronal activity in primary sensory cortices. PFC has been proposed as the source of top-down attention signals that modulate cross-modal processing in primary sensory cortices in favor of the attended features. Studies have highlighted the effects of attention on neuronal responses in primary sensory cortices, such as an increase in neuronal discharges and a decrease in the variability of neuronal responses. The black arrows correspond to the direct connections from PFC to primary sensory cortices. The gray arrows correspond to the top-down modulation of sensory processing in primary sensory cortices during attention. (D) Neural mechanisms of bottom-up and top-down cross-modal processing. (i) Spike trains before and after stimulus. Neuronal firing is random pre-stimulus, whereas post-stimulus firing rate is enhanced and marked by a precisely timed onset. Stimulus is represented by the red arrow. (ii) Phase reset as a mechanism of bottom-up cross-modal processing. The phase of oscillatory activity is random pre-stimulus, but resets post-stimulus. Stimulus is represented by the red arrow. (iii) Phase locking as a mechanism of bottom-up and top-down sensory processing. Black lines on the peak of the ongoing oscillation indicate spikes. Effective communication occurs when spiking activity of area b arrives at the high excitatory phase of area c and induces spikes in area c. Ineffective communication occurs when spiking activity in area b arrives at the low excitation phase of the signal a and fails to induce spikes in area a. When spiking activity in area b arrives at the rising phase of area a, communication between effective and ineffective levels (indicated by crossed-out check mark) occurs. (iv) Communication between two areas using cross-frequency coupling (CFC). Signal b shows that green and gray high frequency rhythms “ride” on the black low frequency rhythm. CFC between signal a and b enables area a and b to communicate through high frequency rhythm (marked in green). CFC between signal b and c enables b and c to communicate through high frequency rhythm.
Bottom-Up Cross-Modal Processing in Primary Sensory Cortices
Sensory interactions have primarily been demonstrated in high-level association cortices, such as PFC or posterior parietal cortex (PPC) (Lippert et al., 2013; Yau et al., 2015; Song et al., 2017). However, cross-modal processing has been shown to take place already at early stages of sensory processing, such as in the brainstem (Aitkin et al., 1981; Jain and Shore, 2006; Koehler et al., 2011), thalamus (Komura et al., 2005; Allen et al., 2017; Bieler et al., 2018) or primary sensory cortices (Lakatos et al., 2007; Kayser et al., 2010; Sieben et al., 2013).
The superior colliculus (SC) of the midbrain received particular attention when investigating the principles of multisensory processing. The SC receives multiple ascending (Edwards et al., 1979; Mize, 1983) and descending (Clemo and Stein, 1984; Meredith and Clemo, 1989) unisensory afferent sources that converge onto individual neurons, making the SC a prime model to study mechanisms of cross-modal processing. Deep-layer multisensory neurons of the SC control sensory as well as motor responses. Cross-modal but not unimodal, or multiple unimodal stimuli of the same modality (Alvarado et al., 2007), cause an enhancement of neuronal firing (Meredith and Stein, 1983; Perrault et al., 2005), which consequently mediates orienting behavior (Stein et al., 1988; Gingras et al., 2009). It has been shown that the inputs from cortical association areas are critical to manifest cross-modal responses in the SC (Stein et al., 2002; Alvarado et al., 2009). While SC neurons in behaving animals continue to respond to multiple sensory modalities following cortical inactivation, multisensory responses are suppressed, and multisensory integration is eliminated (Jiang et al., 2002, 2007).
The described neuronal responses to cross-modal stimuli in first-order thalamic nuclei and primary sensory cortices occur at too short latency to result from processing feedback information. Thus, also low-level putatively unimodal brain areas integrate cross-modal information in a bottom-up manner. The bottom-up detection and discrimination of stimuli are fundamental stages of sensory processing, because they allow, on the one hand, for rapid detection of a stimulus, and on the other hand, for discrimination between similar stimuli based on fine details (Guo et al., 2017). The detection and discrimination of a stimulus are improved when it provides features from multiple modalities (Gleiss and Kayser, 2012; Sheppard et al., 2013; Siemann et al., 2014; Hollensteiner et al., 2015; Nikbakht et al., 2018).
Similar mechanisms of cross-modal processing first described in the cat SC have also been found in rodent SC (Gharaei et al., 2018) as well as in primary sensory cortices, thus challenging the strict hierarchical model of sensory processing (Foxe and Schroeder, 2005). For example, co-presentation of an auditory stimulus enhances orientation selectivity of V1 neurons (Ibrahim et al., 2016). This cross-modal enhancement of neuronal firing was strongest under low-contrast conditions, suggesting that cross-modal information is particularly beneficial for perceptually-guided behavior under ambiguous situations. In addition to cue-integration, cross-modal processing also depends on modality segregation, i.e. the suppression of neuronal activity in one modality-specific primary sensory cortex due to the concurrent presentation of a stimulus of a non-matching sensory modality (Iurilli et al., 2012; Song et al., 2017; Bieler et al., 2018; Gharaei et al., 2018). For example, Gharaei et al. (2018) demonstrated that unisensory stimulation enhances neuronal responses in the SC, whereas cross-modal stimulation rarely enhances but rather suppresses neuronal firing discharges. At the level of primary sensory cortex, Iurilli et al. (2012) showed that evoked activity in A1 enhances local inhibitory firing in deep layers of V1, which in turn decreases the activity of V1 supragranular pyramidal neurons. Consequently, at behavioral level, visually-conditioned responses were suppressed by acoustic stimulation. Experimental research examining the mechanisms of sensory convergence in low-level sensory regions emphasized the processing and relay of basic object feature information (Iurilli et al., 2012; Sieben et al., 2013; Bieler et al., 2018; Morrill and Hasenstaub, 2018). However, the formation, storage, and utilization of cross-modal object representations during behavior require an interaction of neuronal areas accounting for sensory and cognitive processing (Hindley et al., 2014; Reid et al., 2014; Jacklin et al., 2016). Thus, while both sensory integration and separation are part of bottom-up cross-modal processing in primary sensory cortices, the mechanisms underlying the functional communication between low- and high-level brain areas during cross-modal perception are still largely unknown.
Top-Down Modulation of Cross-Modal Processing in Primary Sensory Cortices
Creating a consistent mental representation of the multisensory environment depends on more than the convergence of sensory information in primary sensory cortices (Choi et al., 2018). Sensory processing in primary sensory cortices is modulated top-down to create a multisensory perception, and finally, behavioral action (Ernst and Newell, 2007; Gilbert and Li, 2013; Talsma, 2015; Bizley et al., 2016; Kunicki et al., 2019). In particular, top-down influences from high- to low-level brain areas allow for the preferential processing, and thereby the facilitation of specific sensory inputs in primary sensory cortices (Talsma et al., 2010). Such top-down information may be related to attention, expectation or perceptual demands (Paneri and Gregoriou, 2017; Choi et al., 2018). Attention is a core property of all perceptual and cognitive operations. Given the limited capacity to process competing environmental inputs, attentional mechanisms allow for the selection and modulation as well as for sustained focus on information most relevant for behavior (Chun et al., 2011). Attention modulates neuronal activity and improves the signal-to-noise ratio thereby increasing signal efficacy for attended stimuli and enhancing the representation of attended features (Noudoost et al., 2010). Attention facilitates the integration of multisensory inputs in a top-down manner (Fiebelkorn et al., 2010; Mühlberg and Soto-Faraco, 2019). Top-down modulation enables the flexible selection of information based on task goals, as well as providing an order for selectively modulating multiple stimuli within each modality if they are competing for processing resources (Alsius et al., 2005; Doty et al., 2006). For example, Terreros et al. (2016) showed that mice are able to selectively focus on a visual stimulus, ignoring distractive auditory stimuli during selective attention in a two-choice visual discrimination task (Terreros et al., 2016). Furthermore, top-down modulation reweights sensory information and facilitates the integration of cross-modal inputs (Alsius et al., 2005; Busse et al., 2005; Bresciani and Ernst, 2007; Talsma et al., 2007; Lakatos et al., 2009; Fiebelkorn et al., 2010; Muhlberg et al., 2014). Prior cross-modal exploration of task-relevant objects significantly facilitates the detection performance of a rat in a cross-modal object recognition task (Jacklin et al., 2016). Moreover, rats are able to recognize a visually presented object, which has been only explored by the tactile sense (Winters and Reid, 2010). Top-down task demands further modulate cross-modal processing in primary sensory cortices. For example, during the free exploration of novel objects in the dark (whisker-based tasks), V1 and S1 responses carried comparable amounts of information about object identity (Vasconcelos et al., 2011). However, during the execution of an aperture tactile discrimination task, which is based on top-down task demands, S1 showed faster and more robust tactile recruitment when compared to V1.
Several frontal and parietal cortical regions, such as PPC and PFC, have been proposed as the source of top-down modulatory signals (Noudoost et al., 2010; Winters and Reid, 2010; Jacklin et al., 2016; Paneri and Gregoriou, 2017; Mohan et al., 2018). For example, it has been shown that top-down modulation originating in PPC influences cross-modal processing in primary sensory cortices (Mohan et al., 2017; Kunicki et al., 2019), and damage to PPC leads to performance deficits in sensory discrimination tasks (Binkofski et al., 2001; Winters and Reid, 2010). Given the well-established role of PFC in cognitive control and executive function (Miller and Cohen, 2001), it has been hypothesized that it modulates sensory processing in primary sensory cortices as well (Buschman and Miller, 2007). Bichot et al. (2015) showed in non-human primates performing a visual search task, that feature-based attention adjusts the neural firing activity of prefrontal neurons representing an attended feature to quickly locate a target object (Bichot et al., 2015). Moreover, neural responses in PFC emerge earlier when compared to the responses in visual cortex during covert attention tasks (Gregoriou et al., 2009; Monosov et al., 2010; Zhou and Desimone, 2011; Lennert and Martinez-Trujillo, 2013; Bichot et al., 2015; Siegel et al., 2015). Pharmacological inactivation of PFC induced space-specific impairments in a covert visual search task, and was particularly prominent when a shift in attention was required (Monosov and Thompson, 2009). The PFC might provide top-down modulatory signals to primary sensory cortices through direct axonal projections. For example, Zhang et al. (2014) showed that activation of prefrontal local GABAergic circuits powerfully influences sensory processing in V1 through direct connectivity from PFC to V1 (Zhang et al., 2014). Moreover, prefrontal modulatory signals may reach primary sensory cortices via the sensory thalamus. Stimulating the PFC has been shown to increase tactile responses and alter basal activity in the ventrobasal region of the thalamus (Cao et al., 2008). In line with this, optogenetic manipulation of prefrontal activity perturbs the ability of mice to appropriately select between conflicting visual and auditory stimuli during a cross-modal divided-attention task that is known to depend on prefrontal-thalamic interactions (Wimmer et al., 2015).
Anatomical Substrate of Interactions Between Neuronal Networks Accounting for Sensory and Cognitive Processing
Direct bottom- up (Henschke et al., 2015; Mowery et al., 2016; Bieler et al., 2017b; Henschke et al., 2017) and top-down cortico-cortical (Zhang et al., 2014; Makino and Komiyama, 2015) as well as indirect cortico-thalamo-cortical pathways (Theyel et al., 2010; Roth et al., 2016) represent the anatomical substrate of the functional communication between low- and high-level brain areas during cross-modal processing (Figures 1B,C).
Short latency cross-modal interactions in low-level sensory cortices rely on direct long-range connections (Sieben et al., 2013; Stehberg et al., 2014; Henschke et al., 2015). For example, visual stimulation modulates S1 activity via direct cortico-cortical connections, while pharmacological inactivation of V1 diminishes cross-modal effects in S1 (Sieben et al., 2013). In addition, optogenetic stimulation of A1-V1 projection neurons sharpens the orientation selectivity of neurons in V1 (Ibrahim et al., 2016). Similarly, impairing the direct A1-V1 connectivity by cortico-cortical transections abolishes the sound-driven hyperpolarization of V1 (Iurilli et al., 2012). Compared to the described connectivity patterns between primary sensory cortices in rodents (Burkhalter, 1989; Wang and Burkhalter, 2007; Stehberg et al., 2014; Henschke et al., 2015), direct cortico-cortical projections are sparse in primate primary sensory areas, which has functional implications on cross-modal processing (Falchier et al., 2002; Clavagnier et al., 2004; Cappe and Barone, 2005). Single-cell recordings revealed only subthreshold neuronal responses in primate primary sensory areas (Molholm et al., 2002; Lakatos et al., 2007; Kayser et al., 2008), and suprathreshold multisensory neurons were restricted to higher cortical areas (Fu et al., 2003; Ghazanfar et al., 2005). In contrast to primate low-level areas where feedback cross-modal information only has a subthreshold influence on its postsynaptic targets (Allman et al., 2009), multisensory responses in rodent primary sensory cortices might rely on the direct cortico-cortical connections and less on feedback information from higher cortical association areas. This suggests that the presence or absence of multisensory suprathreshold effects might result from the number and strength of cross-modal inputs reaching rodent or primate primary sensory cortices respectively.
In contrast to the early cross-modal responses in primary sensory cortices, cross-modal effects occurring at longer poststimulus latency may be under the control of feedback information, which is sent via projection neurons from high- to low-level sensory areas (Smith et al., 2010; Banks et al., 2011). Recently, Morrill and Hasenstaub (2018) revealed that a minority of neurons in A1 responds at 40 ms after visual stimulus presentation, exceeding the time delay of monosynaptic information transmission. Inputs from higher sensory cortex, such as secondary visual cortex, might account for the occurrence of visual responses with a long latency in A1 (Bizley et al., 2007; Banks et al., 2011). Information between primary sensory cortices may also be transferred via a cortico-thalamic-cortical route (Hackett et al., 2007; Sherman, 2016). For example, Hackett et al. (2007) showed that thalamic nuclei (first-order medial geniculate complex and higher-order posterior nucleus of thalamus) share anatomical connections with somatosensory as well as with auditory cortex. This cortico-thalamo-cortical pathway might resemble the anatomical substrate of tactile information transfer from somatosensory to auditory cortex through first- as well as higher-order thalamus (Schroeder et al., 2001; Kayser et al., 2005).
Besides anatomical projections from higher sensory cortices, long-range prefrontal projection neurons have been proposed to modulate cross-modal responses in primary sensory cortices (Vaneden et al., 1992; Sellers et al., 2015; Zhang S.Y. et al., 2016). For example, Zhang S.Y. et al. (2016) identified retrogradely labeled neurons in the cingulate sulcus of PFC targeting V1. Furthermore, the anterior cingulate subdivision of PFC shares direct connections with V1, while primary and secondary motor cortices are connected to somatosensory and auditory cortex (Zhang S.Y. et al., 2016). The identified direct long-range projections between PFC and primary sensory cortices might act as anatomical substrate for the functional communication between low- and high-level areas during cross-modal processing. Future studies using virus-assisted circuit mapping and optogenetic manipulations shall unravel the contribution of top-down projections from PFC to primary sensory cortices during cross-modal processing.
Mechanisms of Bottom-Up Cross-Modal Processing in Primary Sensory Cortices
Encoding of information requires coordinated neuronal firing that selectively filters relevant from irrelevant environmental information (Parker and Newsome, 1998; Connor et al., 2004; Harris and Mrsic-Flogel, 2013). Two neural communication codes – rate coding (i.e., changes in the frequency of action potentials) and temporal coding (i.e., changes of spike timing in relationship to the phase of network oscillations) – have been described (Oram et al., 2002; Kayser et al., 2009; Meredith and Allman, 2015). These two coding strategies often occur concurrently (Biederlack et al., 2006; Kayser et al., 2009; Bieler et al., 2017b), and as a result, increase the coding capacity (Tiesinga et al., 2008; Kayser et al., 2009; Figure 1Di). It is hypothesized that rate changes in single neurons code for the discrete properties of a stimulus, whereas temporal coding marks the relatedness of neuronal firing among neurons eventually leading to a coherent perception of the stimulus (Singer, 2009). Studies in the SC have identified two major operating principles of cross-modal processing. First, the more spatially and temporally coincident cross-modal cues appear, the greater is the multisensory enhancement (i.e., an increased neuronal response after cross-modal when compared to unimodal stimulation) (Meredith and Stein, 1983; Wallace et al., 1998). Second, the strength of the unimodal cues defines the magnitude of the cross-modal effect, such that weaker individual sensory stimuli evoke stronger cross-modal effects (inverse effectiveness) (Perrault et al., 2005). These principles of cross-modal integration served as a general guideline for deciphering cross-modal processing mechanisms in low-level sensory areas at single-cell and network level (Bieler et al., 2017b; Bieler et al., 2018).
Oscillatory activity reflects the rhythmic excitability fluctuations of neuronal populations within particular frequency bands that correspond to specific spatial scales of brain operation. This rhythmic nature of neural activity creates time windows during which inputs are more effective in driving the neurons. By making use of anatomical connectivity between and within brain networks, neuronal network oscillations account for local-global neuronal interactions as well as for maintaining persistent activity (e.g., during behavioral state) (Buzsaki and Draguhn, 2004; Buzsaki, 2010; Buzsaki and Wang, 2012). Synchronization of neuronal network oscillations subserves neuronal communication and enables the integration of sensory information across distant locations of the brain (Senkowski et al., 2008). Selective communication among neural networks might be achieved by coherence of oscillatory firing patterns (sending neurons) and gain modulation (receiving neurons) (Fries, 2015). Thus, rhythmic synchronization generates sequences of excitation and inhibition which focus the spike output of firing neurons and sensitivity to synaptic inputs of receiving neurons to a short temporal window.
Synchrony of activity in distant neural networks ultimately leads to the binding of anatomically segregated functional networks (Fries, 2005; Canolty et al., 2010; Canolty and Knight, 2010). Since unisensory networks encode relationships between detected information by synchronizing their activity, it raises the likelihood that similar mechanisms are involved in cross-modal processing. For example, information processing by one modality can enhance the population synchrony in lower-order regions responsive to another modality, such as primary sensory cortices or subcortical regions, in reciprocal relationship with other brain regions (Kayser and Logothetis, 2007; Driver and Noesselt, 2008; Tyll et al., 2011). This cross-modal synchrony enhancement of neuronal activity has been described for evoked as well as for induced responses: the impact of an external stimulus sensed by one modality is strengthened by appropriately timed information about the event in another modality (Figure 1Di; Sieben et al., 2013). Furthermore, the phase reset of spontaneous neuronal oscillations might facilitate the communication of distant neural networks during cross-modal processing (Figure 1Dii). The re-alignment of phases of ongoing neuronal oscillations in one processing region in relation to a cue of another sensory modality allows inputs to arrive at a high excitability phase (Lakatos et al., 2007; Kayser et al., 2008; Iurilli et al., 2012; Sieben et al., 2013; Figure 1Diii). In addition, the interaction of oscillations in different frequency bands, termed cross-frequency coupling (CFC), has been proposed as another mechanism of how distant brain regions synchronize their activity to interact (Canolty and Knight, 2010; Figure 1Div). The question arises whether CFC acts as a mechanism for the interaction of multiple sensory areas, and thus the integration of cross-modal inputs in rodent sensory cortices (Canolty et al., 2006; Schroeder and Lakatos, 2009). Recently, we examined the oscillatory interactions underlying CFC in a thalamo-cortical circuit during cross-modal processing (Bieler et al., 2018). Our study revealed a significant increase in beta-gamma phase-amplitude CFC between first-order thalamus and primary somatosensory cortex during cross-modal but not unimodal processing. Thus, the phase of the beta rhythm controls the power of coupled gamma oscillations through synchronization of the gamma amplitude envelope with the beta phase during cross-modal processing in thalamo-cortical networks.
While cross-modal effects at functional and anatomical level are widespread in primary sensory cortices, the exact configuration of a cross-modal stimulus ultimately defines which processing strategy, i.e., enhancement or depression of neural responses, is applied (Meijer et al., 2017).
Mechanisms of Top-Down Modulation of Cross-Modal Processing in Primary Sensory Cortices
Several mechanisms of prefrontal top-down modulation of cross-modal processing in primary sensory cortices have been proposed (Tomita et al., 1999; Barceló et al., 2000; Monosov et al., 2011; Gilbert and Li, 2013; Teufel and Nanay, 2017). Temporal coding of neuronal excitability reflected by oscillatory activity in primary cortices might provide a temporal window for effective processing of top-down information (Figure 1Diii). Phase locking of oscillatory activity between PFC and primary sensory cortices was proposed to fulfill this role. In particular, oscillatory activity in primary sensory cortices creates temporal windows during which top-down PFC signals are more effective in driving neuronal activities in primary cortices during sensory processing. If this holds true, spikes from PFC arriving within temporal excitation windows of the sensory cortices might produce postsynaptic spikes in primary sensory cortices more effectively.
Several studies reported enhanced gamma synchronization between prefrontal and unisensory cortices during attention tasks. For example, Gregoriou et al. (2009) found a specific enhancement in gamma phase synchronization between frontal cortex and V4 during sustained attention in a covert spatial attention task (Gregoriou et al., 2009). Frontal locking of spikes to gamma activity in visual cortex encodes the attended location. Interestingly, frontal spike activity occurred ∼10 ms before the maximal excitability in visual cortex. This time delay might correspond to the transmission lag from frontal cortex to V4. Furthermore, the authors applied Granger causality analysis to study the directional coupling between PFC and V4. They showed that during the early stage of the task, when attention must to be shifted to a relevant location, frontal cortex initiated the oscillatory coupling across PFC and V4. Enhanced phase locking to gamma rhythm in V4 during the attention task was restricted to visual processing neurons, and did not include V4 neurons representing aspects such as visuo-movement or movement (Gregoriou et al., 2012). Of note, the gamma coherence between two distant brain regions may have an artifactual origin. It has been proposed that gamma coherence might reflect the coupling of two phase-locked network oscillations as well as the co-modulating effect of an upstream network common to both recorded networks (Buzsáki and Schomburg, 2015).
According to a largely accepted hypothesis, the PFC selectively facilitates the selection of task relevant information and enhances the representation of attended stimuli in primary sensory cortices (Baluch and Itti, 2011). To address this, Ardid et al. (2010) built a simulated model with weak coupling between two networks resembling a low-level sensory and a high-level brain area (Ardid et al., 2010). Enhanced gamma coupling between these two regions heavily influenced the synchronization between specific neurons encoding attended features across the areas. The results support the idea that inter-areal LFP coupling between PFC and primary sensory cortex selectively facilitates the communication between neurons encoding attention-related information. Several lines of evidence support the hypothesis that the top-down prefrontal signal effectively influences sensory processing in primary cortices. For instance, top-down attention affects V1 processing by enhancing the firing rate of neurons representing the attended stimulus (Treue and Trujillo, 1999; Bichot et al., 2005) and reducing the variability of inter-neuronal correlation (Cohen and Maunsell, 2009; Mitchell et al., 2009; Herrero et al., 2013). The reduced variability of correlation among neurons improves the signal-to-noise ratio for attention-relevant information and promotes efficient coding of attended features. Consequently, the signal-to-noise ratio improves (Cohen and Maunsell, 2009; Mitchell et al., 2009). Moreover, top-down attention modulates local oscillatory activity of primary sensory cortices in a frequency-specific manner (Gregoriou et al., 2015). For example, during attentional selection, neurons in visual and frontal areas encoding the attended location or feature synchronize their activity in gamma frequency (30–60 Hz) range (Tallon-Baudry et al., 2004; Bichot et al., 2005; Fries, 2005; Kreiter et al., 2005; Fries et al., 2008; Gregoriou et al., 2009). This might facilitate the propagation of information between these two areas (Salinas and Sejnowski, 2001; Azouz and Gray, 2003; Fries, 2005, 2009). In addition, reduced local alpha-beta oscillatory activity in V2 and V4 during an attention task (Thut et al., 2006; Fries et al., 2008; Siegel et al., 2008; Gregoriou et al., 2009; Buffalo et al., 2011) has been proposed to inhibit distracting inputs (Palva and Palva, 2007; Händel et al., 2011). Top-down attention also modulates the size and position of visual receptive fields, bursting activity, response latency as well as feature tuning of neurons (Murray and Wojciulik, 2004; David et al., 2008).
Investigation of local circuits and synaptic processes provide additional evidence for top-down modulation of cross-modal processing. Zhang et al. (2014) demonstrated that long-range glutamatergic projections from PFC modulate local circuits in V1 (Zhang et al., 2014). Optogenetic activation of prefrontal neurons led to enhanced responses of V1 neurons. Light stimulation of prefrontal axonal terminals in V1 induced center-surround modulation, which increased the response at the activation site, while suppressing the response at a nearby location. Three subtypes of interneurons in local visual circuits were targeted by top-down prefrontal modulation. First, somatostatin-positive interneurons (SOM+) were critical for surround suppression, since they inhibited the response of pyramidal neurons to the prefrontal input within a 200 μm radius. Second, vasoactive intestinal peptide-positive interneurons (VIP+) were crucial for center facilitation in V1 (Fu et al., 2014), mediating the disinhibition of pyramidal neurons. This disinhibition effect was mainly localized at the site of prefrontal axons in V1 and caused the increase of attention-inducing firing rate. Third, parvalbumin-positive (PV+) GABAergic interneurons were required for long distance inhibition, since their inactivation reduced prefrontal axon-induced inhibitory inputs at a distance of 400 μm. Thus, long-range prefrontal projections act through local microcircuits to exert top-down modulation of sensory processing.
The Emergence of Sensory-Cognitive Interplay During Cross-Modal Development
The brain’s ability to create a coherent perception of the environment by integrating information of various sensory modalities is not present immediately following birth. The development of cross-modal integrative capabilities is a protracted process both in rodents (Ghoshal et al., 2011; Mowery et al., 2016; Hattori and Hensch, 2017) as well as in humans (Scheier et al., 2003; Lewkowicz and Ghazanfar, 2009; Lewkowicz, 2010). This process depends on the alteration and refinement of neural circuitry following uni- and cross-modal sensory experiences.
Cross-modal abilities mature under the influence of intrinsic (i.e., genetic cues) and extrinsic (i.e., environment) factors (Rauschecker et al., 1992; Yu et al., 2010; Frangeul et al., 2016; Moreno-Juan et al., 2017). During embryonic development, molecular cues and genetic programs control the generation, migration, and differentiation of neurons as well as the formation of rudimentary connectivity (Toda et al., 2013; Diao et al., 2018; Telley et al., 2019). At later stages, but before the onset of sensory transduction, spontaneous electrical activity occurring in distinct spatial and temporal patterns refine rudimentary connectivity and facilitate the formation of sensory maps (Galli and Maffei, 1988; Dehorter et al., 2012; Luhmann et al., 2016; Anton-Bolanos et al., 2019). The patterns of spontaneous network activity are conserved across species, and their perturbation causes deficits in network refinement (Huberman et al., 2008). During defined developmental periods (i.e., critical/sensitive periods) the circuits, and later behavioral abilities, are particularly prone to being shaped by experience-dependent electrical activity (Chapman, 2000; Chang and Merzenich, 2003; Pfeiffenberger et al., 2006; Ghoshal et al., 2009; Khazipov et al., 2013). The patterns of electrical activity are similar in age-matched rodents and humans (Khazipov and Luhmann, 2006).
Development of the Tactile System
By using their highly sensitive whiskers, nocturnal rodents can acquire tactile information and build spatial representations of the environment (Petersen, 2007). Whisker-related inputs are processed in somatotopic maps where each whisker is represented by a discrete anatomical unit (“barrel”). Barrel-like cell aggregates form soon after birth (Jhaveri et al., 1991; Schlaggar and O’Leary, 1994). Early sensory experience is mandatory for the development of somatosensory processing. Neonatal whisker trimming from birth on impairs the dendritic complexity of neurons in the barrel cortex and behavioral performance in the gap-crossing task during adulthood (Carvell and Simons, 1996; Lee et al., 2009). Whisker-dependent exploratory behavior does not develop until the second postnatal week (Welker, 1964; Figures 2A,B). This suggests that prior to experience-dependent plasticity other mechanisms must contribute to the development of somatosensory perception. Transcription factors, such as Gbx2, Mash1, and Pax6 have been reported to be involved in pathfinding of axons from thalamus to S1 (Tuttle et al., 1999; Hevner et al., 2002). In addition, discontinuous electrical activity, which appears within the first two postnatal weeks, shapes the development of topographic organization in S1. Several patterns of neonatal electrical activity have been characterized, such as gamma oscillations, spindle bursts with frequencies in theta-beta range, and long-oscillations (Yang et al., 2009; Minlebaev et al., 2011; Yang et al., 2016). Peripheral inputs are not mandatory for the emergence of these early activity patterns. Gamma oscillations and spindle bursts remain after the peripheral pathways were lesioned (Khazipov et al., 2004; Minlebaev et al., 2011; Yang et al., 2013). Early activity patterns may act as a template for the emergence of cortical topography. For instance, the volume of synchronized neurons during spindle burst activity reflects the anatomical size of the future barrels (Yang et al., 2016). Long oscillations are assumed to synchronize large neuronal networks and boost the formation of functional neuronal ensembles (Yang et al., 2009). With ongoing maturation, rodents start to whisker and early tactile experience further refines the somatosensory circuits.
Figure 2. Schematic diagram displaying the developmental milestones of sensory and limbic development in rodents. (A) Schematic illustration displaying the developmental timeline of sensory development from postnatal day (P) zero onward. (B) Schematic arrows showing the time points (marked by dotted line) of (i) the critical/sensitive period of somatosensory (green), auditory (blue), and visual (red) development, (ii) the onset of unisensory behavior, and (iii) the start of cross-modal modulation. Uni- and cross-modal inputs in the first days of life are hypothesized to drive the development of the limbic system in a bottom-up manner, while bottom-up as well as top-down interactions between the primary sensory cortices and limbic system are present at later stages of development (gray boxes, bottom). (C) Same as (B) for PFC. Time points shown in gray arrow mark developmental milestones of limbic system development.
Development of the Auditory System
Similar to tactile development, the maturation of auditory pathways containing orderly representations of frequency selectivity involves both molecular cues and spontaneous electrical activity. For example, neurotrophins such as BDNF and NT-3, ephrins (Ernfors et al., 1992; Hossain et al., 2008) and semaphorins (Gu et al., 2003; Webber and Raz, 2006) have been reported to guide auditory innervation. Spontaneous electrical activity further refines and maintains the tonotopic architecture set by molecular cues (Wang and Bergles, 2015). In rodents, the ability to respond to acoustic stimuli emerges around postnatal (P) day 12 (Uziel et al., 1981; Kelly, 1992; Figures 2A,B). Experience-dependent activity then promotes the fine-tuning of auditory networks (Friauf and Lohmann, 1999). Before this age, environmental factors regulate the maturation of auditory processes. For example, early interactions with the mother modulate the maturation of the auditory system in pups (Cárdenas et al., 2015). Auditory reflexes in pups were accelerated when the mothers were reared in an enriched environment during gestation. Moreover, exposure to frequency-enriched acoustic environments during the first 14 days after birth significantly decreased the threshold of auditory responses in a frequency-specific manner (Chang et al., 2018). Rearing in a disturbed acoustic environment impairs the development of the auditory system (Zhang et al., 2002; Chang and Merzenich, 2003; Nakahara et al., 2004; Speechley et al., 2007). Early noise exposure induced permanent structural changes in the rat auditory system (Ouda et al., 2016). Rat pups exposed to trains of 5 kHz pure tones showed larger regions of auditory cortex tuned to 5 kHz at adulthood (Han et al., 2007). Thus, over-representations of certain frequencies during early development likely reduces auditory discrimination.
Development of the Visual System
Rodents are born blind. The retina starts to be light-sensitive during the second postnatal week, and shortly after that, the eyelids open (Figures 2A,B; Sernagor, 2005). From birth on, axonal projections from the lateral geniculate nucleus (LGN) target cells in the granular layers of V1 leading to the initiation of cortical topographical organization. During early development, when the retina is light-insensitive, bursts of action potentials (i.e., retinal waves) emerge under the control of the cholinergic system (Brombas et al., 2017) and propagate across the retina (Wong et al., 1993). These retinal waves are transmitted via the optic nerve to the LGN and finally to V1, where they boost cortical spindle bursts (Hanganu et al., 2006). At each developmental stage of V1, retinal waves differ in their properties, thereby instructing the development of visual feature processing mechanisms (Huberman et al., 2008).
With the onset of light sensitivity, visual experience shapes the cortical topography (Smith and Trachtenberg, 2007). In cats, monocular visual deprivation led to a size reduction of columns corresponding to the sutured eye, whereas columns corresponding to the non-deprived eye expanded (Hubel et al., 1977; Le Vay et al., 1980). Visual deprivation during the sensitive period leads to alterations in thalamo-cortical connectivity (Fox and Wong, 2005; Hofer et al., 2008) and as a consequence alters the input organization from both eyes (Espinosa and Stryker, 2012). Experience has been shown to control the tuning of V1 neurons to stimulus orientation and direction (Hubel and Wiesel, 1962; Weliky et al., 1996). Thus, even though coarse orientation selectivity emerges under the influence of experience-independent neuronal activity (White et al., 2001), high-level orientation selectivity appears only in the presence of visual inputs (Chapman and Stryker, 1993). In contrast, neither molecular cues nor spontaneous activity, but visual experience seems to be required for tuning V1 neurons to stimulus direction (Li et al., 2006).
External Inputs Controlling the Development of Cross-Modal Processing in Rodent Primary Sensory Cortex
While a wealth of studies documented the relevance of early electrical activity for the maturation of topographic organization, few studies addressed the mechanisms of cross-modal development in primary sensory cortices. One key question is whether perturbing unisensory development – even prior to full responsiveness of all stimulus-related sensory modalities – has long-lasting consequences for the development of cross-modal processing. It appears that cross-modal development requires a certain level of unisensory maturity (Ghoshal et al., 2011; Sieben et al., 2015). For instance, Sieben et al. (2015) showed that tactile deprivation shortly after birth (P0-5) causes abnormal visual-tactile cross-modal processing later in life. Furthermore, it has been shown that the power and phase of neuronal activity were modulated by cross-modal stimuli of juvenile rats with only minimal cross-modal experience (i.e., closed eye lids, but light-sensitive retina and tactile sensation in P14-16 rats) (Bieler et al., 2017a). Thus, network interactions ensuring cross-modal processing emerge before cross-modal experience and refine during juvenile development (Figure 2B).
Development of the PFC
As previously mentioned, the PFC is involved in memory, attention, and decision-making (Miller, 2000; Vertes, 2006). In addition, it is considered to act as a hub of cross-modal processing (Fuster et al., 2000; Nieder, 2017). Overall, the PFC follows the developmental milestones described for primary sensory cortices. Early patterns of oscillatory activity are highly discontinuous and temporally fragmented (Brockmann et al., 2011), yet they emerge a few days later when compared with V1 or S1. Moreover, the maturation of the PFC is remarkably prolonged when compared to other cortical areas (Leipsic, 1901; van Eden and Uylings, 1985). The prefrontal cytoarchitecture and correspondingly, the executive and mnemonic abilities, are not fully developed until adolescence (van Eden and Uylings, 1985).
The functional development of PFC seems to be controlled by activity in the intermediate/ventral hippocampus (HP). Hippocampal theta bursts emerging a few days before prefrontal spindle bursts, drive the generation of neonatal prefrontal oscillations by phase-locking the neuronal firing via axonal pathways (Brockmann et al., 2011). Remarkably, the early entrainment of prefrontal-hippocampal networks is critical for the mnemonic ontogeny at juvenile stage (Krüger et al., 2012). During later development (∼P10), the oscillatory activity in both PFC and hippocampus switches from discontinuous bursts to continuous theta-gamma oscillations. This switch occurs almost simultaneously in the prefrontal and primary sensory cortices (Colonnese and Khazipov, 2010).
Sensory-Cognitive Interactions During Development
As outlined in sections “Development of the Tactile System,” “Development of the Auditory System,” and “Development of the Visual System,” early endogenous and sensory-driven activity patterns contribute to the development and refinement of neuronal networks (Hanganu et al., 2006; Minlebaev et al., 2009; Yang et al., 2009; Yang et al., 2013). Perturbing sensory inputs during critical/sensitive periods of development has profound effects on the neuronal activity and its underlying anatomical connectivity, and thus affects behavior (Fagiolini et al., 1994; Carvell and Simons, 1996; Erzurumlu and Gaspar, 2012; Levelt and Hubener, 2012; Kral, 2013).
Perturbation of a sensory input leads to anatomical and functional modifications in the remaining sensory systems. As a consequence, neurons adaptively reorganize to integrate the function of other sensory systems, in a process termed cross-modal plasticity (Bavelier and Neville, 2002; Lee and Whitt, 2015). Cross-modal plasticity alters perceptual abilities. For example, several studies have shown that bilateral lid suture or enucleation impairs orientation and direction selectivity of V1 neurons, but enhances the processing of auditory and somatosensory inputs in V1 (Rauschecker et al., 1992; Rauschecker and Kniepert, 1994; Yaka et al., 2000; Izraeli et al., 2002). Similar cross-modal activation patterns after sensory deprivation have been observed in other primary sensory cortices (Goel et al., 2006; Hunt et al., 2006; Lee and Whitt, 2015; Meng et al., 2015).
Recently, the effects of non-visual inputs on experience-dependent plasticity in V1 during early postnatal development have been investigated (Hattori and Hensch, 2017; Figure 2B). Concurrent visual-auditory inputs impaired the development of orientation selectivity of V1 neurons if they occurred before or after the critical period. However, the effect was dampened if cross-modal visual-auditory stimuli occurred during the critical period. The authors suggest that this effect is likely caused by a sound-driven balance of suppression and enhancement of V1 spiking activity, which is required for the tuning and consolidation of visual selectivity. Similarly, it has been shown that the onset of visual experience controls the development of auditory processing (Mowery et al., 2016). In particular, the critical period of auditory development was precociously closed by early eyelid opening and extended by delayed eyelid opening (Figure 2B).
Few experimental data have documented the impact of altering the functional anatomy and neuronal activity of primary sensory cortices on the development of PFC (Kolb and Gibb, 2015). It has been shown that sensory deprivation increases the density of interneurons in PFC (Ueno et al., 2015). This is in line with findings from primary visual cortex where the laminar distribution of PV+ neurons is altered following enucleation (Desgent et al., 2010). Overall, a mechanistic understanding of the effects of sensory deprivation on the bidirectional communication between primary sensory cortices and PFC is currently lacking.
As discussed in section “External Inputs Controlling the Development of Cross-Modal Processing in Rodent Primary Sensory Cortex,” perturbations of unisensory development prior to full maturation of all unisensory systems has long-lasting consequences for the development of cross-modal processing abilities (Ghoshal et al., 2011; Sieben et al., 2015). Notably, during the sensitive period of tactile development, the functional maturation of the PFC is boosted by the excitatory drive from the hippocampus (Brockmann et al., 2011; Bitzenhofer et al., 2017; Ahlbeck et al., 2018; Figure 2C). However, it is largely unknown how early sensory development affects the maturation of the limbic system. Several studies have shown that sensory experience is important for synaptic pruning during PFC development (Schanberg and Field, 1987; Richards et al., 2012). For example, raising rodents in a tactile-enriched environment from birth on increases the prefrontal spine density and improves the performance in PFC-dependent tasks at adulthood. The increased dendritic branching and spine density in PFC (Kolb et al., 2012; Kolb and Gibb, 2015) argue for significant plastic changes occurring when experiencing a sensory enriched environment. Thus, sensory-driven activity might directly impact the maturation of the limbic system.
Early electrical activity in sensory and limbic circuits may facilitate the network development required for their communication (Mohns and Blumberg, 2008). Neocortical spindle bursts are induced by proprioceptive feedback which is initiated by twitches of the distal limbs (Khazipov et al., 2004). These spindle bursts drive the activation of CA1 neurons and critically depend on neocortical-hippocampal interactions (Mohns and Blumberg, 2010). Since myoclonic movements induce bursts of activity in the medial entorhinal cortex, which in turn drives hippocampal responses, it has been suggested that entorhinal-hippocampal interactions are part of a large-scale bottom-up circuit activated during neonatal movements (Valeeva et al., 2019). While the impact of somatosensory processing on limbic system development began to be elucidated, it is currently unknown whether similar bottom-up interactions exist for other sensory systems. Similarly, the impact of top-down PFC activity on early sensory development and its importance for adult cross-modal processing capabilities are still unknown.
Animal and Human Research as Background for Brain-Inspired Intelligent Robotics
Neuroscientific insights can be harnessed to build adaptive and intelligent machines. Given recent advances in calcium (Ca2+) imaging using genetically encoded Ca2+ indicators and in the use of optogenetic tools for causal manipulation of neural circuits (Fenno et al., 2011; Grienberger and Konnerth, 2012), current and future research can provide a plethora of insights into the neuronal computations of cross-modal processing. Based on brain-like neural architectures and biologically plausible learning mechanisms (Pitti et al., 2009), computer implementations can create robot perception and action (Floreano et al., 2014). The field of robotics is one of the most dynamic areas of technological development (Zhang B. et al., 2016), and robots performing very specific tasks are increasingly found in industry, service, and medicine. A growing field is also the interplay between robotics and neuroscience. For instance, equipping cognitive robots with the ability to process and integrate cross-modal information streams ensures that they will interact with the environment more efficiently, even under conditions of sensory uncertainty (Parisi et al., 2019). Similarly, developmental robotics, which is motivated by human cognitive and behavioral development, aims to provide a better understanding of the development of cognitive processes using robots with rich sensory and motor capabilities as testing platforms (Breazeal and Scassellati, 2002; Lungarella et al., 2003; Prince, 2008; Cangelosi and Schlesinger, 2015, 2018).
As outlined above, low-level sensory and high-level neural networks accounting for cognitive processing interact in a bottom-up and top-down manner to create a coherent perception of the multisensory environment. Similarly, bottom-up and top-down processing underlying the integration of multipleisensory information streams play a crucial role in the development of autonomous agents and cognitive robots. However, these two research streams often developed independently. Closer interactions between them appear mutually beneficial for several reasons. First, biological inspiration for the modeling of bottom-up cross-modal processing in robots is of crucial interest in order to endow agents with improved robustness, flexibility and performance, particularly in the case of uncertain, ambiguous or incongruent cross-modal inputs (Parisi et al., 2019). For example, biological inspiration has played a major role in the field of odor-guided navigation (Russell, 2001). Bailey et al. (2005) developed a robot with multisensory processing capabilities, and in particular stellar odor-tracking performance similar to that found in animals, in order to locate the source of chemical plumes (Bailey et al., 2005). Barsky et al. (2019) applied a deep learning method to combine disparate sensory inputs, such as auditory and visual information. Cross-modal processing facilitated the learning of a humanoid drumming robot to generate suitable motion sequences to match desired unseen audio or video sequences (Barsky et al., 2019). Axenie et al. (2016) proposed a novel audio-visual sensory processing architecture for robust multisensory fusion in robotic systems, which is inspired by the distributed macro-architecture of the mammalian cortex (Axenie et al., 2016).
Second, biological inspiration for the modeling of top-down cross-modal processing in robots is mandatory for autonomous agents and cognitive robots to develop perception through active groping. Fujimoto et al. (2009) developed a robot being able to pick up dishes based on active groping. The robot roughly formulated a strategy for selecting dishes placed close to each other. Subsequently, by actively acquiring the geometric information of the dishes during the implementation of the strategy, the robot was able to efficiently complete the task (Fujimoto et al., 2009). Inoue (1971) developed a robot to search for a block by actively moving the hand along a predefined track and detecting contact with items using touch sensors (Inoue, 1971). Maekawa et al. (1992) developed a finger-shaped tactile sensor which could reconstruct the shape of an object by actively moving along a predefined grid and detecting the position and direction of contact by using sensors (Maekawa et al., 1992). These studies demonstrate that robots have the capability to progressively learn in an ever-changing multisensory environment by means of self-exploration and social interaction.
However, robots are still limited in their dynamic movements, emotional perception and adaptive interactions with humans, and this drawback limits their application (Wiese et al., 2017; Cross et al., 2019). To overcome this challenge, brain-inspired intelligent robotics may equip systems with advanced human-like cognitive abilities such as improved multisensory processing and learning capabilities by mimicking the structures and mechanisms underlying sensory-cognitive processing (section “Sensory-Cognitive Interplay During Cross-Modal Processing”). In fact, multisensory perception has been named as one of the key sensory-cognitive functions in order for cognitive robots to thrive in a complex and dynamic environment (Zhang B. et al., 2016). A lack of multisensory perceptive capabilities, makes it more sophisticated to acquire other cognitive computations and to function autonomously. Continuous learning of robotic systems is crucial, because internal models of the multisensory world must be acquired and adapted throughout development in order for multisensory processing capabilities to emerge (section “The Emergence of Sensory-Cognitive Interplay During Cross-Modal Development”) (Rohlf et al., 2017). Recent endeavors led to the creation of an open source humanoid called NICO (Neuro- Inspired COmpanion), which due to its flexible design and open and modular hardware and software framework can adapt to individual experimental set-ups and opens the door to multimodal human-robot interaction research with the aim of developing autonomous agents and cognitive robots (Kerzel et al., 2017).
Conclusion and Future Lines of Research
It has been hypothesized that the bottom-up sensory drive contributes to establishing neuronal circuits in the limbic system during early development (Mohns and Blumberg, 2008). At adulthood, the interaction between low-level sensory and high-level limbic areas enables cross-modal perceptual decision-making. Cross-modal representations are transferred from primary sensory cortices to PFC in a bottom-up manner, and the representation of an attended stimulus in primary sensory cortices is selectively enhanced by top-down prefrontal modulation (Bizley et al., 2016). However, the interactions between primary sensory cortices and PFC during bottom-up/top-down cross-modal processing have been poorly characterized. To this end, techniques that specifically manipulate neuronal pathways between PFC and primary sensory cortices are necessary. Relying on recent advances in optogenetic terminal field excitation/inhibition, selectively illuminating axon terminals originating from PFC and targeting primary sensory cortices, would allow for the manipulation of the direct pathways between PFC and primary sensory cortices. This pathway-specific targeting will link function and connectivity underlying cross-modal processing within sensory-limbic circuits.
Author Contributions
XX, IH-O, and MB contributed to the conception and design of the study, and wrote sections of the manuscript. XX and MB organized the literature data base and wrote the first draft of the manuscript. All authors contributed to manuscript revision, read and approved the submitted version.
Funding
This work was funded by grants from the European Research Council (ERC-2015-CoG 681577 to IH-O) and the German Research Foundation (Ha 4466/10-1, SPP 1665, SFB 936 B5 to IH-O). IH-O is a founding member of FENS Kavli Network of Excellence.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Ahlbeck, J., Song, L., Chini, M., Bitzenhofer, S. H., and Hanganu-Opatz, I. L. (2018). Glutamatergic drive along the septo-temporal axis of hippocampus boosts prelimbic oscillations in the neonatal mouse. eLife 7:e33158 doi: 10.7554/eLife.33158
Aitkin, L. M., Kenyon, C. E., and Philpott, P. (1981). The representation of the auditory and somatosensory systems in the external nucleus of the cat inferior colliculus. J. Comp. Neurol. 196, 25–40. doi: 10.1002/cne.901960104
Allen, A. E., Procyk, C. A., Brown, T. M., and Lucas, R. J. (2017). Convergence of visual and whisker responses in the primary somatosensory thalamus (ventral posterior medial region) of the mouse. J. Physiol. 595, 865–881. doi: 10.1113/JP272791
Allman, B. L., Keniston, L. P., and Meredith, M. A. (2009). Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing. Brain Topogr. 21, 157–167. doi: 10.1007/s10548-009-0088-3
Alsius, A., Navarra, J., Campbell, R., and Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Curr. Biol. 15, 839–843. doi: 10.1016/j.cub.2005.03.046
Alvarado, J. C., Stanford, T. R., Rowland, B. A., Vaughan, J. W., and Stein, B. E. (2009). Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 29, 6580–6592. doi: 10.1523/JNEUROSCI.0525-09.2009
Alvarado, J. C., Vaughan, J. W., Stanford, T. R., and Stein, B. E. (2007). Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205. doi: 10.1152/jn.00018.2007
Anton-Bolanos, N., Sempere-Ferrandez, A., Guillamon-Vivancos, T., Martini, F. J., Perez-Saiz, L., Gezelius, H., et al. (2019). Prenatal activity from thalamic neurons governs the emergence of functional cortical maps in mice. Science 364:987. doi: 10.1126/science.aav7617
Ardid, S., Wang, X.-J., Gomez-Cabrero, D., and Compte, A. (2010). Reconciling coherent oscillation with modulation of irregular spiking activity in selective attention: gamma-range synchronization between sensory and executive cortical areas. J. Neurosci. 30, 2856–2870. doi: 10.1523/JNEUROSCI.4222-09.2010
Axenie, C., Richter, C., and Conradt, J. (2016). A self-synthesis approach to perceptual learning for multisensory fusion in robotics. Sensors 16:1751 doi: 10.3390/s16101751
Azouz, R., and Gray, C. M. (2003). Adaptive coincidence detection and dynamic gain control in visual cortical neurons in vivo. Neuron 37, 513–523. doi: 10.1016/s0896-6273(02)01186-8
Bailey, J. K., Willis, M. A., and Quinn, R. D. (2005). “A multi-sensory robot for testing biologically-inspired odor plume tracking strategies,” in Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics. Tokyo.
Baluch, F., and Itti, L. (2011). Mechanisms of top-down attention. Trends Neurosci. 34, 210–224. doi: 10.1016/j.tins.2011.02.003
Banks, M. I., Uhlrich, D. J., Smith, P. H., Krause, B. M., and Manning, K. A. (2011). Descending projections from extrastriate visual cortex modulate responses of cells in primary auditory cortex. Cereb. Cortex 21, 2620–2638. doi: 10.1093/cercor/bhr048
Barceló, F., Suwazono, S., and Knight, R. T. (2000). Prefrontal modulation of visual processing in humans. Nat. Neurosci. 3, 399–403. doi: 10.1038/73975
Barsky, A., Zito, C., Mori, H., Ogata, T., and Wyatt, J. (2019). Multisensory learning framework for robot drumming. arXiv [Preprint]. Available at: https://www2.informatik.uni-hamburg.de/wtm/WorkshopCLIR18/index.php (accessed November 1, 2019).
Bavelier, D., and Neville, H. J. (2002). Cross-modal plasticity: where and how? Nat. Rev. Neurosci. 3, 443–452. doi: 10.1038/nrn848
Bichot, N. P., Heard, M. T., DeGennaro, E. M., and Desimone, R. (2015). A source for feature-based attention in the prefrontal cortex. Neuron 88, 832–844. doi: 10.1016/j.neuron.2015.10.001
Bichot, N. P., Rossi, A. F., and Desimone, R. (2005). Parallel and serial neural mechanisms for visual search in macaque area V4. Science 308, 529–534. doi: 10.1126/science.1109676
Biederlack, J., Castelo-Branco, M., Neuenschwander, S., Wheeler, D. W., Singer, W., and Nikolic, D. (2006). Brightness induction: rate enhancement and neuronal synchronization as complementary codes. Neuron 52, 1073–1083. doi: 10.1016/j.neuron.2006.11.012
Bieler, M., Sieben, K., Cichon, N., Schildt, S., Roder, B., and Hanganu-Opatz, I. L. (2017a). Rate and temporal coding convey multisensory information in primary sensory cortices. eNeuro 4:ENEURO.0037-17.2017 doi: 10.1523/ENEURO.0037-17.2017
Bieler, M., Sieben, K., Schildt, S., Roder, B., and Hanganu-Opatz, I. L. (2017b). Visual-tactile processing in primary somatosensory cortex emerges before cross-modal experience. Synapse 71:e21958. doi: 10.1002/syn.21958
Bieler, M., Xu, X., Marquardt, A., and Hanganu-Opatz, I. L. (2018). Multisensory integration in rodent tactile but not visual thalamus. Sci. Rep. 8:15684.
Binkofski, F., Kunesch, E., Classen, J., Seitz, R. J., and Freund, H. J. (2001). Tactile apraxia: unimodal apractic disorder of tactile object exploration associated with parietal lobe lesions. Brain 124, 132–144.
Bitzenhofer, S. H., Ahlbeck, J., Wolff, A., Wiegert, J. S., Gee, C. E., Oertner, T. G., et al. (2017). Layer-specific optogenetic activation of pyramidal neurons causes beta-gamma entrainment of neonatal networks. Nat. Commun. 8:14563. doi: 10.1038/ncomms14563
Bizley, J. K., Jones, G. P., and Town, S. M. (2016). Where are multisensory signals combined for perceptual decision-making? Curr. Opin. Neurobiol. 40, 31–37. doi: 10.1016/j.conb.2016.06.003
Bizley, J. K., Nodal, F. R., Bajo, V. M., Nelken, I., and King, A. J. (2007). Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb. Cortex 17, 2172–2189. doi: 10.1093/cercor/bhl128
Breazeal, C., and Scassellati, B. (2002). Robots that imitate humans. Trends Cogn. Sci. 6, 481–487. doi: 10.1016/s1364-6613(02)02016-8
Bresciani, J. P., and Ernst, M. (2007). Signal reliability modulates auditory-tactile integration for event counting. Neuroreport 18, 1157–1161. doi: 10.1097/wnr.0b013e3281ace0ca
Brockmann, M. D., Poschel, B., Cichon, N., and Hanganu-Opatz, I. L. (2011). Coupled oscillations mediate directed interactions between prefrontal cortex and hippocampus of the neonatal rat. Neuron 71, 332–347. doi: 10.1016/j.neuron.2011.05.041
Brombas, A., Kalita-de Croft, S., Cooper-Williams, E. J., and Williams, S. R. (2017). Dendro-dendritic cholinergic excitation controls dendritic spike initiation in retinal ganglion cells. Nat. Commun. 8:15683. doi: 10.1038/ncomms15683
Buffalo, E. A., Fries, P., Landman, R., Buschman, T. J., and Desimone, R. (2011). Laminar differences in gamma and alpha coherence in the ventral stream. PNAS 108, 11262–11267. doi: 10.1073/pnas.1011284108
Burkhalter, A. (1989). Intrinsic connections of rat primary visual cortex: laminar organization of axonal projections. J. Comp. Neurol. 279, 171–186. doi: 10.1002/cne.902790202
Buschman, T. J., and Miller, E. K. (2007). Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science 315, 1860–1862. doi: 10.1126/science.1138071
Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H., and Woldorff, M. G. (2005). The spread of attention across modalities and space in a multisensory object. PNAS 102, 18751–18756. doi: 10.1073/pnas.0507704102
Buzsaki, G. (2010). Neural syntax: cell assemblies, synapsembles, and readers. Neuron 68, 362–385. doi: 10.1016/j.neuron.2010.09.023
Buzsaki, G., and Draguhn, A. (2004). Neuronal oscillations in cortical networks. Science 304, 1926–1929. doi: 10.1126/science.1099745
Buzsáki, G., and Schomburg, E. W. (2015). What does gamma coherence tell us about inter-regional neural communication? Nat. Neurosci. 18, 484–489. doi: 10.1038/nn.3952
Buzsaki, G., and Wang, X. J. (2012). Mechanisms of gamma oscillations. Annu. Rev. Neurosci. 35, 203–225. doi: 10.1146/annurev-neuro-062111-150444
Calvert, G. A., and Thesen, T. (2004). Multisensory integration: methodological approaches and emerging principles in the human brain. J. Physiol. Paris 98, 191–205. doi: 10.1016/j.jphysparis.2004.03.018
Cangelosi, A., and Schlesinger, M. (2015). Developmental Robotics: From Babies to Robots. Cambridge, MA: MIT Press.
Cangelosi, A., and Schlesinger, M. (2018). From Babies to robots: the contribution of developmental robotics to developmental psychology. Child Dev. Perspect. 12, 183–188. doi: 10.1111/cdep.12282
Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Kirsch, H. E., et al. (2006). High gamma power is phase-locked to theta oscillations in human neocortex. Science 313, 1626–1628. doi: 10.1126/science.1128115
Canolty, R. T., Ganguly, K., Kennerley, S. W., Cadieu, C. F., Koepsell, K., Wallis, J. D., et al. (2010). Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies. Proc. Natl. Acad. Sci. U.S.A. 107, 17356–17361. doi: 10.1073/pnas.1008306107
Canolty, R. T., and Knight, R. T. (2010). The functional role of cross-frequency coupling. Trends Cogn. Sci. 14, 506–515. doi: 10.1016/j.tics.2010.09.001
Cao, X.-H., Wang, D. H., Bai, J., Zhou, S. C., and Zhou, Y. D. (2008). Prefrontal modulation of tactile responses in the ventrobasal thalamus of rats. Neurosci. Lett. 435, 152–157. doi: 10.1016/j.neulet.2008.02.030
Cappe, C., and Barone, P. (2005). Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. Eur. J. Neurosci. 22, 2886–2902. doi: 10.1111/j.1460-9568.2005.04462.x
Cárdenas, L., García-García, F., Santiago-Roque, I., Martínez, A. J., Coria-Ávila, G. A., and Corona-Morales, A. A. (2015). Enriched environment restricted to gestation accelerates the development of sensory and motor circuits in the rat pup. Int. J. Dev. Neurosci. 41, 68–73. doi: 10.1016/j.ijdevneu.2014.11.008
Carvell, G., and Simons, D. (1996). Abnormal tactile experience early in life disrupts active touch. J. Neurosci. 16, 2750–2757. doi: 10.1523/jneurosci.16-08-02750.1996
Chang, A., Chen, P., Guo, S., Xu, N., Pan, W., Zhang, H., et al. (2018). Specific influences of early acoustic environments on cochlear hair cells in postnatal mice. Neural Plast. 2018:13. doi: 10.1155/2018/5616930
Chang, E. F., and Merzenich, M. M. (2003). Environmental noise retards auditory cortical development. Science 300, 498–502. doi: 10.1126/science.1082163
Chapman, B. (2000). Necessity for afferent activity to maintain eye-specific segregation in ferret lateral geniculate nucleus. Science 287, 2479–2482. doi: 10.1126/science.287.5462.2479
Chapman, B., and Stryker, M. (1993). Development of orientation selectivity in ferret visual cortex and effects of deprivation. J. Neurosci. 13, 5251–5262. doi: 10.1523/jneurosci.13-12-05251.1993
Chen, T.-W., Wardill, T. J., Sun, Y., Pulver, S. R., Renninger, S. L., Baohan, A., et al. (2013). Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499, 295–300. doi: 10.1038/nature12354
Choi, I., Lee, J. Y., and Lee, S. H. (2018). Bottom-up and top-down modulation of multisensory integration. Curr. Opin. Neurobiol. 52, 115–122. doi: 10.1016/j.conb.2018.05.002
Chun, M. M., Golomb, J. D., and Turk-Browne, N. B. (2011). A taxonomy of external and internal attention. Annu. Rev. Psychol. 62, 73–101. doi: 10.1146/annurev.psych.093008.100427
Clavagnier, S., Falchier, A., and Kennedy, H. (2004). Long-distance feedback projections to area V1: implications for multisensory integration, spatial awareness, and visual consciousness. Cogn. Affect. Behav. Neurosci. 4, 117–126. doi: 10.3758/cabn.4.2.117
Clemo, H. R., and Stein, B. E. (1984). Topographic organization of somatosensory corticotectal influences in cat. J. Neurophysiol. 51, 843–858. doi: 10.1152/jn.1984.51.5.843
Cohen, M. R., and Maunsell, J. H. R. (2009). Attention improves performance primarily by reducing interneuronal correlations. Nat. Neurosci. 12:1594. doi: 10.1038/nn.2439
Colonnese, M. T., and Khazipov, R. (2010). Slow activity transients in infant rat visual cortex: a spreading synchronous oscillation patterned by retinal waves. J. Neurosci. 30, 4325–4337. doi: 10.1523/JNEUROSCI.4995-09.2010
Connor, C. E., Egeth, H. E., and Yantis, S. (2004). Visual attention: bottom-up versus top-down. Curr. Biol. 14, R850–R852.
Cross, E. S., Hortensius, R., and Wykowska, A. (2019). From social brains to social robots: applying neurocognitive insights to human-robot interaction. Philos. Trans. R. Soc. B 374:20180024. doi: 10.1098/rstb.2018.0024
David, S. V., Hayden, B. Y., Mazer, J. A., and Gallant, J. L. (2008). Attention to stimulus features shifts spectral tuning of V4 neurons during natural vision. Neuron 59, 509–521. doi: 10.1016/j.neuron.2008.07.001
Dehorter, N., Vinay, L., Hammond, C., and Ben-Ari, Y. (2012). Timing of developmental sequences in different brain structures: physiological and pathological implications. Eur. J. Neurosci. 35, 1846–1856. doi: 10.1111/j.1460-9568.2012.08152.x
Desgent, S., Boire, D., and Ptito, M. (2010). Altered expression of Parvalbumin and Calbindin in interneurons within the primary visual cortex of neonatal enucleated hamsters. Neuroscience 171, 1326–1340. doi: 10.1016/j.neuroscience.2010.10.016
Diao, Y. P., Chen, Y. Q., Zhang, P. J., Cui, L. Y., and Zhang, J. Y. (2018). Molecular guidance cues in the development of visual pathway. Protein Cell 9, 909–929. doi: 10.1007/s13238-017-0490-7
Doty, T. J., Woldorff, M. G., and Talsma, D. (2006). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690. doi: 10.1093/cercor/bhk016
Driver, J., and Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57, 11–23. doi: 10.1016/j.neuron.2007.12.013
Edwards, S. B., Ginsburgh, C. L., Henkel, C. K., and Stein, B. E. (1979). Sources of subcortical projections to the superior colliculus in the cat. J. Comp. Neurol. 184, 309–329. doi: 10.1002/cne.901840207
Ernfors, P., Merlio, J. P., and Persson, H. (1992). Cells expressing mRNA for neurotrophins and their receptors during embryonic rat development. Eur. J. Neurosci. 4, 1140–1158. doi: 10.1111/j.1460-9568.1992.tb00141.x
Ernst, M. O., and Newell, F. N. (2007). Multisensory recognition of actively explored objects. Can. J. Exp. Psychol. 61, 242–253. doi: 10.1037/cjep2007025
Erzurumlu, R. S., and Gaspar, P. (2012). Development and critical period plasticity of the barrel cortex. Eur. J. Neurosci. 35, 1540–1553. doi: 10.1111/j.1460-9568.2012.08075.x
Espinosa, J. S., and Stryker, M. P. (2012). Development and plasticity of the primary visual cortex. Neuron 75, 230–249. doi: 10.1016/j.neuron.2012.06.009
Fagiolini, M., Pizzorusso, T., Berardi, N., Domenici, L., and Maffei, L. (1994). Functional postnatal-development of the rat primary visual-cortex and the role of visual experience - dark rearing and monocular deprivation. Vis. Res. 34, 709–720. doi: 10.1016/0042-6989(94)90210-0
Falchier, A., Clavagnier, S., Barone, P., and Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22, 5749–5759. doi: 10.1523/jneurosci.22-13-05749.2002
Fenno, L., Yizhar, O., and Deisseroth, K. (2011). The development and application of optogenetics. Annu. Rev. Neurosci. 34, 389–412. doi: 10.1146/annurev-neuro-061010-113817
Fiebelkorn, I. C., Foxe, J. J., and Molholm, S. (2010). Dual mechanisms for the cross-sensory spread of attention: how much do learned associations matter? Cereb. Cortex 20, 109–120. doi: 10.1093/cercor/bhp083
Floreano, D., Ijspeert, A. J., and Schaal, S. (2014). Robotics and neuroscience. Curr. Biol. 24, R910–R920. doi: 10.1016/j.cub.2014.07.058
Fox, K., and Wong, R. O. L. (2005). A comparison of experience-dependent plasticity in the visual and somatosensory systems. Neuron 48, 465–477. doi: 10.1016/j.neuron.2005.10.013
Foxe, J., and Schroeder, C. (2005). The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423. doi: 10.1097/00001756-200504040-00001
Frangeul, L., Pouchelon, G., Telley, L., Lefort, S., Luscher, C., and Jabaudon, D. (2016). A cross-modal genetic framework for the development and plasticity of sensory pathways. Nature 538, 96–98. doi: 10.1038/nature19770
Friauf, E., and Lohmann, C. (1999). Development of auditory brainstem circuitry. Cell Tissue Res. 297, 187–195. doi: 10.1007/s004410051346
Fries, P. (2005). A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends Cogn. Sci. 9, 474–480. doi: 10.1016/j.tics.2005.08.011
Fries, P. (2009). Neuronal Gamma-band synchronization as a fundamental process in cortical computation. Annu. Rev. Neurosci. 32, 209–224. doi: 10.1146/annurev.neuro.051508.135603
Fries, P. (2015). Rhythms for cognition: communication through coherence. Neuron 88, 220–235. doi: 10.1016/j.neuron.2015.09.034
Fries, P., Womelsdorf, T., Oostenveld, R., and Desimone, R. (2008). The effects of visual stimulation and selective visual attention on rhythmic neuronal synchronization in macaque area V4. J. Neurosci. 28, 4823–4835. doi: 10.1523/JNEUROSCI.4499-07.2008
Fu, K.-M. G., Johnston, T. A., Shah, A. S., Arnold, L., Smiley, J., Hackett, T. A., et al. (2003). Auditory cortical neurons respond to somatosensory stimulation. J. Neurosci. 23, 7510–7515. doi: 10.1523/jneurosci.23-20-07510.2003
Fu, Y., Jason Tucciarone, M., Espinosa, J. S., Sheng, N., Daniel Darcy, P., Roger Nicoll, A., et al. (2014). A cortical circuit for gain control by behavioral state. Cell 156, 1139–1152. doi: 10.1016/j.cell.2014.01.050
Fujimoto, J., Mizuuchi, I., Sodeyama, Y., Yamamoto, K., Muramatsu, N., Ohta, S., et al. (2009). “Picking up dishes based on active groping with multisensory robot hand,” in Proceedings of the The 18th IEEE International Symposium on Robot and Human Interactive Communication. Toyama.
Fuster, J. M., Bodner, M., and Kroger, J. K. (2000). Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405, 347–351. doi: 10.1038/35012613
Galli, L., and Maffei, L. (1988). Spontaneous impulse activity of rat retinal Ganglion-cells in prenatal life. Science 242, 90–91. doi: 10.1126/science.3175637
Gharaei, S., Arabzadeh, E., and Solomon, S. G. (2018). Integration of visual and whisker signals in rat superior colliculus. Sci. Rep. 8:16445. doi: 10.1038/s41598-018-34661-8
Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., and Logothetis, N. K. (2005). Multisensory integration of dynamic faces and voices in Rhesus monkey auditory cortex. J. Neurosci. 25, 5004–5012. doi: 10.1523/jneurosci.0799-05.2005
Ghoshal, A., Pouget, P., Popescu, M., and Ebner, F. (2009). Early bilateral sensory deprivation blocks the development of coincident discharge in rat barrel cortex. J. Neurosci. 29, 2384–2392. doi: 10.1523/JNEUROSCI.4427-08.2009
Ghoshal, A., Tomarken, A., and Ebner, F. (2011). Cross-sensory modulation of primary sensory cortex is developmentally regulated by early sensory experience. J. Neurosci. 31, 2526–2536. doi: 10.1523/JNEUROSCI.5547-10.2011
Gilbert, C. D., and Li, W. (2013). Top-down influences on visual processing. Nat. Rev. Neurosci. 14, 350–363. doi: 10.1038/nrn3476
Gingras, G., Rowland, B. A., and Stein, B. E. (2009). The differing impact of multisensory and unisensory integration on behavior. J. Neurosci. 29, 4897–4902. doi: 10.1523/JNEUROSCI.4120-08.2009
Gleiss, S., and Kayser, C. (2012). Audio-visual detection benefits in the rat. PLoS One 7:e45677. doi: 10.1371/journal.pone.0045677
Goel, A., Jiang, B., Xu, L. W., Song, L., Kirkwood, A., and Lee, H. K. (2006). Cross-modal regulation of synaptic AMPA receptors in primary sensory cortices by visual experience. Nat. Neurosci. 9, 1001–1003. doi: 10.1038/nn1725
Goodman, C. S., and Shatz, C. J. (1993). Developmental mechanisms that generate precise patterns of neuronal connectivity. Cell 72, 77–98. doi: 10.1016/s0092-8674(05)80030-3
Gregoriou, G. G., Gotts, S. J., and Desimone, R. (2012). Cell-type-specific synchronization of neural activity in FEF with V4 during attention. Neuron 73, 581–594. doi: 10.1016/j.neuron.2011.12.019
Gregoriou, G. G., Gotts, S. J., Zhou, H., and Desimone, R. (2009). High-frequency, long-range coupling between prefrontal and visual cortex during attention. Science 324, 1207–1210. doi: 10.1126/science.1171402
Gregoriou, G. G., Paneri, S., and Sapountzis, P. (2015). Oscillatory synchrony as a mechanism of attentional processing. Brain Res. 1626, 165–182. doi: 10.1016/j.brainres.2015.02.004
Grienberger, C., and Konnerth, A. (2012). Imaging calcium in neurons. Neuron 73, 862–885. doi: 10.1016/j.neuron.2012.02.011
Gu, C., Rodriguez, E. R., Reimert, D. V., Shu, T., Fritzsch, B., Richards, L. J., et al. (2003). Neuropilin-1 conveys Semaphorin and VEGF signaling during neural and cardiovascular development. Dev. Cell 5, 45–57. doi: 10.1016/s1534-5807(03)00169-2
Guo, W., Clause, A. R., Barth-Maron, A., and Polley, D. B. (2017). A Corticothalamic circuit for dynamic switching between feature detection and discrimination. Neuron 95:180-194.e5. doi: 10.1016/j.neuron.2017.05.019
Hackett, T. A., De La Mothe, L. A., Ulbert, I., Karmos, G., Smiley, J., and Schroeder, C. E. (2007). Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. J. Comp. Neurol. 502, 924–952. doi: 10.1002/cne.21326
Hammond-Kenny, A., Bajo, V. M., King, A. J., and Nodal, F. R. (2017). Behavioural benefits of multisensory processing in ferrets. Eur. J. Neurosci. 45, 278–289. doi: 10.1111/ejn.13440
Han, Y. K., Köver, H., Insanally, M. N., Semerdjian, J. H., and Bao, S. (2007). Early experience impairs perceptual discrimination. Nat. Neurosci. 10:1191. doi: 10.1038/nn1941
Händel, B. F., Haarmeier, T., and Jensen, O. (2011). Alpha Oscillations correlate with the successful inhibition of unattended stimuli. J. Cogn. Neurosci. 23, 2494–2502. doi: 10.1162/jocn.2010.21557
Hanganu, I. L., Ben-Ari, Y., and Khazipov, R. (2006). Retinal waves trigger spindle bursts in the neonatal rat visual cortex. J. Neurosci. 26, 6728–6736. doi: 10.1523/jneurosci.0752-06.2006
Harris, K. D., and Mrsic-Flogel, T. D. (2013). Cortical connectivity and sensory coding. Nature 503, 51–58. doi: 10.1038/nature12654
Hattori, R., and Hensch, T. K. (2017). Developmental dynamics of cross-modality in mouse visual cortex. bioRxiv [Preprint]. doi: 10.1101/150847
Henschke, J. U., Noesselt, T., Scheich, H., and Budinger, E. (2015). Possible anatomical pathways for short-latency multisensory integration processes in primary sensory cortices. Brain Struct. Funct. 220, 955–977. doi: 10.1007/s00429-013-0694-4
Henschke, J. U., Oelschlegel, A. M., Angenstein, F., Ohl, F. W., Goldschmidt, J., Kanold, P. O., et al. (2017). Early sensory experience influences the development of multisensory thalamocortical and intracortical connections of primary sensory cortices. Brain Struct. Funct. 223, 1165–1190. doi: 10.1007/s00429-017-1549-1
Herrero, J. L., Marc Gieselmann, A., Sanayei, M., and Thiele, A. (2013). Attention-induced variance and noise correlation reduction in macaque V1 is mediated by NMDA receptors. Neuron 78, 729–739. doi: 10.1016/j.neuron.2013.03.029
Hevner, R. F., Miyashita-Lin, E., and Rubenstein, J. L. R. (2002). Cortical and thalamic axon pathfinding defects in Tbr1, Gbx2, and Pax6 mutant mice: evidence that cortical and thalamic axons interact and guide each other. J. Comp. Neurol. 447, 8–17. doi: 10.1002/cne.10219
Hindley, E. L., Nelson, A. J. D., Aggleton, J. P., and Vann, S. D. (2014). Dysgranular retrosplenial cortex lesions in rats disrupt cross-modal object recognition. Learn Mem. 21, 171–179. doi: 10.1101/lm.032516.113
Hofer, S. B., Mrsic-Flogel, T. D., Bonhoeffer, T., and Hübener, M. (2008). Experience leaves a lasting structural trace in cortical circuits. Nature 457:313. doi: 10.1038/nature07487
Hollensteiner, K. J., Pieper, F., Engler, G., Konig, P., and Engel, A. K. (2015). Crossmodal integration improves sensory detection thresholds in the ferret. PLoS One 10:e0124952. doi: 10.1371/journal.pone.0124952
Hossain, W. A., D’Sa, C., and Morest, D. K. (2008). Interactive roles of fibroblast growth factor 2 and neurotrophin 3 in the sequence of migration, process outgrowth, and axonal differentiation of mouse cochlear ganglion cells. J. Neurosci. Res. 86, 2376–2391. doi: 10.1002/jnr.21685
Hubel, D. H., and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160, 106–154. doi: 10.1113/jphysiol.1962.sp006837
Hubel, D. H., Wiesel, T. N., LeVay, S., Barlow, H. B., and Gaze, R. M. (1977). Plasticity of ocular dominance columns in monkey striate cortex. Philos. Trans. Royal Soc. B 278, 377–409. doi: 10.1098/rstb.1977.0050
Huberman, A. D., Feller, M. B., and Chapman, B. (2008). Mechanisms underlying development of visual maps and receptive fields. Annu. Rev. Neurosci. 31, 479–509. doi: 10.1146/annurev.neuro.31.060407.125533
Hunt, D. L., Yamoah, E. N., and Krubitzer, L. (2006). Multisensory plasticity in congenitally deaf mice: how are cortical areas functionally specified? Neuroscience 139, 1507–1524. doi: 10.1016/j.neuroscience.2006.01.023
Ibrahim, L. A., Mesik, L., Ji, X. Y., Fang, Q., Li, H. F., Li, Y. T., et al. (2016). Cross-modality sharpening of visual cortical processing through layer-1-mediated inhibition and disinhibition. Neuron 89, 1031–1045. doi: 10.1016/j.neuron.2016.01.027
Inoue, H. (1971). Computer controlled bilateral manipulator. Bull. JSME 14, 199–207. doi: 10.1299/jsme1958.14.199
Iurilli, G., Ghezzi, D., Olcese, U., Lassi, G., Nazzaro, C., Tonini, R., et al. (2012). Sound-driven synaptic inhibition in primary visual cortex. Neuron 73, 814–828. doi: 10.1016/j.neuron.2011.12.026
Izraeli, R., Koay, G., Lamish, M., Heicklen-Klein, A. J., Heffner, H. E., Heffner, R. S., et al. (2002). Cross-modal neuroplasticity in neonatally enucleated hamsters: structure, electrophysiology and behaviour. Eur. J. Neurosci. 15, 693–712. doi: 10.1046/j.1460-9568.2002.01902.x
Jacklin, D. L., Cloke, J. M., Potvin, A., Garrett, I., and Winters, B. D. (2016). The dynamic multisensory engram: neural circuitry underlying crossmodal object recognition in rats changes with the nature of object experience. J. Neurosci. 36, 1273–1289. doi: 10.1523/JNEUROSCI.3043-15.2016
Jain, R., and Shore, S. (2006). External inferior colliculus integrates trigeminal and acoustic information: unit responses to trigeminal nucleus and acoustic stimulation in the guinea pig. Neurosci. Lett. 395, 71–75. doi: 10.1016/j.neulet.2005.10.077
Jhaveri, S., Erzurumlu, R. S., and Crossin, K. (1991). Barrel construction in rodent neocortex: role of thalamic afferents versus extracellular matrix molecules. PNAS 88, 4489–4493. doi: 10.1073/pnas.88.10.4489
Jiang, W., Jiang, H., Rowland, B. A., and Stein, B. E. (2007). Multisensory orientation behavior is disrupted by neonatal cortical ablation. J. Neurophysiol. 97, 557–562. doi: 10.1152/jn.00591.2006
Jiang, W., Jiang, H., and Stein, B. E. (2002). Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255. doi: 10.1162/089892902760807230
Kayser, C., and Logothetis, N. K. (2007). Do early sensory cortices integrate cross-modal information? Brain Struct. Funct. 212, 121–132. doi: 10.1007/s00429-007-0154-0
Kayser, C., Logothetis, N. K., and Panzeri, S. (2010). Visual enhancement of the information representation in auditory cortex. Curr. Biol. 20, 19–24. doi: 10.1016/j.cub.2009.10.068
Kayser, C., Montemurro, M. A., Logothetis, N. K., and Panzeri, S. (2009). Spike-phase coding boosts and stabilizes information carried by spatial and temporal Spike patterns. Neuron 61, 597–608. doi: 10.1016/j.neuron.2009.01.008
Kayser, C., Petkov, C. I., Augath, M., and Logothetis, N. K. (2005). Integration of touch and sound in auditory cortex. Neuron 48, 373–384. doi: 10.1016/j.neuron.2005.09.018
Kayser, C., Petkov, C. I., and Logothetis, N. K. (2008). Visual modulation of neurons in auditory cortex. Cereb. Cortex 18, 1560–1574. doi: 10.1093/cercor/bhm187
Keil, J., and Senkowski, D. (2018). Neural Oscillations Orchestrate Multisensory Processing. Neuroscientist 24, 609–626. doi: 10.1177/1073858418755352
Kelly, J. B. (1992). Behavioral Development of the Auditory Orientation Response. Development of Auditory and Vestibular Systems. Amsterdam: Elsevier, 391–418.
Kerzel, M., Strahl, E., Magg, S., Navarro-Guerrero, N., Heinrich, S., et al. (2017). “NICO — Neuro-inspired companion: a developmental humanoid robot platform for multimodal interaction,” in Proceeding of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). Portugal.
Khazipov, R., and Luhmann, H. J. (2006). Early patterns of electrical activity in the developing Cereb Cortex of humans and rodents. Trends Neurosci. 29, 414–418. doi: 10.1016/j.tins.2006.05.007
Khazipov, R., Minlebaev, M., and Valeeva, G. (2013). Early gamma Oscillations. Neuroscience 250, 240–252. doi: 10.1016/j.neuroscience.2013.07.019
Khazipov, R., Sirota, A., Leinekugel, X., Holmes, G. L., Ben-Ari, Y., and Buzsáki, G. (2004). Early motor activity drives spindle bursts in the developing somatosensory cortex. Nature 432, 758–761. doi: 10.1038/nature03132
Koehler, S. D., Pradhan, S., Manis, P. B., and Shore, S. E. (2011). Somatosensory inputs modify auditory spike timing in dorsal cochlear nucleus principal cells. Eur. J. Neurosci. 33, 409–420. doi: 10.1111/j.1460-9568.2010.07547.x
Kolb, B., and Gibb, R. (2015). Plasticity in the prefrontal cortex of adult rats. Front. Cell Neurosci. 9:15. doi: 10.3389/fncel.2015.00015
Kolb, B., Mychasiuk, R., Muhammad, A., Li, Y. L., Frost, D. O., and Gibb, R. (2012). Experience and the developing prefrontal cortex. PNAS 109, 17186–17193. doi: 10.1073/pnas.1121251109
Komura, Y., Tamura, R., Uwano, T., Nishijo, H., and Ono, T. (2005). Auditory thalamus integrates visual inputs into behavioral gains. Nat. Neurosci. 8, 1203–1209. doi: 10.1038/nn1528
Kral, A. (2013). Auditory critical periods: a review from system’s perspective. Neuroscience 247, 117–133. doi: 10.1016/j.neuroscience.2013.05.021
Kreiter, A. K., Taylor, K., Mandon, S., and Freiwald, W. A. (2005). Coherent Oscillatory activity in monkey area V4 predicts successful allocation of attention. Cereb. Cortex 15, 1424–1437. doi: 10.1093/cercor/bhi023
Krüger, H.-S., Brockmann, M. D., Salamon, J., Ittrich, H., and Hanganu-Opatz, I. L. (2012). Neonatal hippocampal lesion alters the functional maturation of the prefrontal cortex and the early cognitive development in pre-juvenile rats. Neurobiol. Learn. Mem. 97, 470–481. doi: 10.1016/j.nlm.2012.04.001
Kunicki, C., Moioli, R. C., Pais-Vieira, M., Salles Cunha Peres, A., Morya, E., et al. (2019). Frequency-specific coupling in fronto-parieto-occipital cortical circuits underlie active tactile discrimination. Sci. Rep. 9:5105. doi: 10.1038/s41598-019-41516-3
Lakatos, P., Chen, C. M., O’Connell, M. N., Mills, A., and Schroeder, C. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292. doi: 10.1016/j.neuron.2006.12.011
Lakatos, P., O’Connell, M. N., Barczak, A., Mills, A., Javitt, D. C., and Schroeder, C. E. (2009). The leading sense: supramodal control of neurophysiological context by attention. Neuron 64, 419–430. doi: 10.1016/j.neuron.2009.10.014
Le Vay, S., Wiesel, T. N., and Hubel, D. H. (1980). The development of ocular dominance columns in normal and visually deprived monkeys. J. Comp. Neurol. 191, 1–51. doi: 10.1002/cne.901910102
Lee, H. K., and Whitt, J. L. (2015). Cross-modal synaptic plasticity in adult primary sensory cortices. Curr. Opin. Neurol. 35, 119–126. doi: 10.1016/j.conb.2015.08.002
Lee, L.-J., Chen, W. J., Chuang, Y. W., and Wang, Y. C. (2009). Neonatal whisker trimming causes long-lasting changes in structure and function of the somatosensory system. Exp. Neurol. 219, 524–532. doi: 10.1016/j.expneurol.2009.07.012
Leipsic, P. F. O. (1901). Develomental (myelogenetic) localisation of the cereb cortex in the human subject. Lancet 158, 1027–1030. doi: 10.1016/s0140-6736(01)01429-5
Lennert, T., and Martinez-Trujillo, J. C. (2013). Prefrontal neurons of opposite spatial preference display distinct target selection dynamics. J. Neurosci. 33, 9520–9529. doi: 10.1523/JNEUROSCI.5156-12.2013
Levelt, C. N., and Hubener, M. (2012). Critical-period plasticity in the visual cortex. Annu. Rev. Neurosci. 35, 309–330. doi: 10.1146/annurev-neuro-061010-113813
Lewkowicz, D. J. (2010). Infant perception of audio-visual speech synchrony. Dev. Psychol. 46, 66–77. doi: 10.1037/a0015579
Lewkowicz, D. J., and Ghazanfar, A. A. (2009). The emergence of multisensory systems through perceptual narrowing. Trends Cogn. Sci. 13, 470–478. doi: 10.1016/j.tics.2009.08.004
Li, Y., Fitzpatrick, D., and White, L. E. (2006). The development of direction selectivity in ferret visual cortex requires early visual experience. Nat. Neurosci. 9, 676–681. doi: 10.1038/nn1684
Lippert, M. T., Takagaki, K., Kayser, C., and Ohl, F. W. (2013). Asymmetric multisensory interactions of visual and Somatosensory responses in a region of the rat parietal cortex. PLoS One 8:e0063631. doi: 10.1371/journal.pone.0063631
Luhmann, H. J., Sinning, A., Yang, J. W., Reyes-Puerta, V., Stuttgen, M. C., Kirischuk, S., et al. (2016). Spontaneous neuronal activity in developing neocortical networks: from single cells to large-scale interactions. Front. Neural Circ. 10:40. doi: 10.3389/fncir.2016.00040
Lungarella, M., Metta, G., Pfeifer, R., and Sandini, G. (2003). Developmental robotics: a survey. Connect. Sci. 15, 151–190. doi: 10.1080/09540090310001655110
Maekawa, H., Tanie, K., Komoriya, K., Kaneko, M., Horiguchi, C., et al. (1992). “Development of a finger-shaped tactile sensor and its evaluation by active touch,” in Proceedings of the IEEE International Conference on Robotics and Automation. San Francisco, CA.
Makino, H., and Komiyama, T. (2015). Learning enhances the relative impact of top-down processing in the visual cortex. Nat. Neurosci. 18, 1116–1122. doi: 10.1038/nn.4061
Meijer, G. T., Montijn, J. S., Pennartz, C. M. A., and Lansink, C. S. (2017). Audiovisual modulation in mouse primary visual cortex depends on cross-modal stimulus configuration and congruency. J. Neurosci. 37, 8783–8796. doi: 10.1523/JNEUROSCI.0468-17.2017
Meijer, G. T., Pie, J. L., Dolman, T. L., Pennartz, C. M. A., and Lansink, C. S. (2018). Audiovisual integration enhances stimulus detection performance in mice. Front. Behav. Neurosci. 12:231. doi: 10.3389/fnbeh.2018.00231
Meng, X., Kao, J. P., Lee, H. K., and Kanold, P. O. (2015). Visual deprivation causes refinement of intracortical circuits in the auditory cortex. Cell Rep. 12, 955–964. doi: 10.1016/j.celrep.2015.07.018
Meredith, M. A., and Allman, B. L. (2015). Single-unit analysis of somatosensory processing in the core auditory cortex of hearing ferrets. Eur. J. Neurosci. 41, 686–698. doi: 10.1111/ejn.12828
Meredith, M. A., and Clemo, H. R. (1989). Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study. J. Comp. Neurol. 289, 687–707. doi: 10.1002/cne.902890412
Meredith, M. A., and Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391. doi: 10.1126/science.6867718
Miller, E. K. (2000). The prefontral cortex and cognitive control. Nat. Rev. Neurosci. 1, 59–65. doi: 10.1038/35036228
Miller, E. K., and Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202. doi: 10.1146/annurev.neuro.24.1.167
Minlebaev, M., Ben-Ari, Y., and Khazipov, R. (2009). NMDA receptors pattern early activity in the developing barrel cortex in vivo. Cereb. Cortex 19, 688–696. doi: 10.1093/cercor/bhn115
Minlebaev, M., Colonnese, M., Tsintsadze, T., Sirota, A., and Khazipov, R. (2011). Early gamma Oscillations synchronize developing Thalamus and cortex. Science 334, 226–229. doi: 10.1126/science.1210574
Mitchell, J. F., Sundberg, K. A., and Reynolds, J. H. (2009). Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4. Neuron 63, 879–888. doi: 10.1016/j.neuron.2009.09.013
Mize, R. R. (1983). Patterns of convergence and divergence of retinal and cortical synaptic terminals in the cat superior colliculus. Exp. Brain Res. 51, 88–96.
Mohan, H., de Haan, R., Mansvelder, H. D., and de Kock, C. P. J. (2017). The posterior parietal cortex as integrative hub for whisker sensorimotor information. Neuroscience 368, 240–245. doi: 10.1016/j.neuroscience.2017.06.020
Mohan, H., Gallero-Salas, Y., Carta, S., Sacramento, J., Laurenczy, B., Sumanovski, L. T., et al. (2018). Sensory representation of an auditory cued tactile stimulus in the posterior parietal cortex of the mouse. Sci. Rep. 8:7739. doi: 10.1038/s41598-018-25891-x
Mohns, E. J., and Blumberg, M. S. (2008). Synchronous bursts of neuronal activity in the developing hippocampus: modulation by active sleep and association with emerging gamma and theta rhythms. J. Neurosci. 28, 10134–10144. doi: 10.1523/JNEUROSCI.1967-08.2008
Mohns, E. J., and Blumberg, M. S. (2010). Neocortical activation of the Hippocampus during sleep in infant rats. J. Neurosci. 30, 3438–3449. doi: 10.1523/JNEUROSCI.4832-09.2010
Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., and Foxe, J. J. (2002). Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn. Brain Res. 14, 115–128. doi: 10.1016/s0926-6410(02)00066-6
Monosov, I. E., Sheinberg, D. L., and Thompson, K. G. (2010). Paired neuron recordings in the prefrontal and inferotemporal cortices reveal that spatial selection precedes object identification during visual search. PNAS 107, 13105–13110. doi: 10.1073/pnas.1002870107
Monosov, I. E., Sheinberg, D. L., and Thompson, K. G. (2011). The effects of prefrontal cortex inactivation on object responses of single neurons in the inferotemporal cortex during visual search. J. Neurosci. 31, 15956–15961. doi: 10.1523/JNEUROSCI.2995-11.2011
Monosov, I. E., and Thompson, K. G. (2009). Frontal eye field activity enhances object identification during covert visual search. J. Neurophysiol. 102, 3656–3672. doi: 10.1152/jn.00750.2009
Moreno-Juan, V., Filipchuk, A., Anton-Bolanos, N., Mezzera, C., Gezelius, H., Andres, B., et al. (2017). Prenatal thalamic waves regulate cortical area size prior to sensory processing. Nat. Commun. 8:14172. doi: 10.1038/ncomms14172
Morrill, R. J., and Hasenstaub, A. R. (2018). Visual information present in infragranular layers of mouse auditory cortex. J. Neurosci. 38, 2854–2862. doi: 10.1523/JNEUROSCI.3102-17.2018
Mowery, T. M., Kotak, V. C., and Sanes, D. H. (2016). The onset of visual experience gates auditory cortex critical periods. Nat. Commun. 7:10416. doi: 10.1038/ncomms10416
Muhlberg, S., Oriolo, G., and Soto-Faraco, S. (2014). Cross-modal decoupling in temporal attention. Eur. J. Neurosci. 39, 2089–2097. doi: 10.1111/ejn.12563
Mühlberg, S., and Soto-Faraco, S. (2019). Cross-modal decoupling in temporal attention between audition and touch. Psychol. Res. 83, 1626–1639. doi: 10.1007/s00426-018-1023-6
Murray, S. O., and Wojciulik, E. (2004). Attention increases neural selectivity in the human lateral occipital complex. Nat. Neurosci. 7, 70–74. doi: 10.1038/nn1161
Nakahara, H., Zhang, L. I., and Merzenich, M. M. (2004). Specialization of primary auditory cortex processing by sound exposure in the critical period. PNAS 101, 7170–7174. doi: 10.1073/pnas.0401196101
Nieder, A. (2017). Magnitude codes for cross-modal working memory in the primate frontal association cortex. Front Neurosci 11:202. doi: 10.3389/fnins.2017.00202
Nikbakht, N., Tafreshiha, A., Zoccolan, D., and Diamonds, M. E. (2018). Supralinear and Supramodal integration of visual and Tactile signals in rats: psychophysics and neuronal mechanisms. Neuron 97, 626–639. doi: 10.1016/j.neuron.2018.01.003
Noudoost, B., Chang, M. H., Steinmetz, N. A., and Moore, T. (2010). Top-down control of visual attention. Curr. Opin. Neurol. 20, 183–190.
Oram, M. W., Xiao, D., Dritschel, B., and Payne, K. R. (2002). The temporal resolution of neural codes: does response latency have a unique role? Philos. Trans. R. Soc. B 357, 987–1001. doi: 10.1098/rstb.2002.1113
Ouda, L., Burianová, J., Balogová, Z., Lu, H. P., and Syka, J. (2016). Structural changes in the adult rat auditory system induced by brief postnatal noise exposure. Brain Struct. Funct. 221, 617–629. doi: 10.1007/s00429-014-0929-z
Palva, S., and Palva, J. M. (2007). New vistas for alpha-frequency band oscillations. Trends Neurosci. 30, 150–158. doi: 10.1016/j.tins.2007.02.001
Paneri, S., and Gregoriou, G. G. (2017). Top-down control of visual attention by the prefrontal cortex. functional specialization and long-range interactions. Front. Neurosci. 11:545. doi: 10.3389/fnins.2017.00545
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019). Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71. doi: 10.1016/j.neunet.2019.01.012
Parker, A. J., and Newsome, W. T. (1998). Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227–277. doi: 10.1146/annurev.neuro.21.1.227
Perrault, T. J., Vaughan, W., Stein, B. E., and Wallace, M. T. (2005). Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586. doi: 10.1152/jn.00926.2004
Petersen, C. C. H. (2007). The functional organization of the barrel cortex. Neuron 56, 339–355. doi: 10.1016/j.neuron.2007.09.017
Pfeiffenberger, C., Yamada, J., and Feldheim, D. A. (2006). Ephrin-As and patterned retinal activity act together in the development of topographic maps in the primary visual system. J. Neurosci. 26, 12873–12884. doi: 10.1523/jneurosci.3595-06.2006
Pitti, A., Alirezaei, H., and Kuniyoshi, Y. (2009). Cross-modal and scale-free action representations through enaction. Neural Netw. 22, 144–154. doi: 10.1016/j.neunet.2009.01.007
Prince, C. G. (2008). Special issue on developmental robotics: can experiments with machines inform theory in infant development? Infant. Child Dev. 17, 1–5. doi: 10.1002/icd.540
Rauschecker, J. P., and Kniepert, U. (1994). Auditory localization behavior in visually deprived cats. Eur. J. Neurosci. 6, 149–160. doi: 10.1111/j.1460-9568.1994.tb00256.x
Rauschecker, J. P., Tian, B., Korte, M., and Egert, U. (1992). Crossmodal changes in the somatosensory vibrissa/barrel system of visually deprived animals. PNAS 89, 5063–5067. doi: 10.1073/pnas.89.11.5063
Reid, J. M., Jacklin, D. L., and Winters, B. D. (2014). Delineating prefrontal cortex region contributions to crossmodal object recognition in rats. Cereb. Cortex 24, 2108–2119. doi: 10.1093/cercor/bht061
Richards, S., Mychasiuk, R., Kolb, B., and Gibb, R. (2012). Tactile stimulation during development alters behaviour and neuroanatomical organization of normal rats. Behav. Brain Res. 231, 86–91. doi: 10.1016/j.bbr.2012.02.043
Rohlf, S., Habets, B., von Frieling, M., and Roder, B. (2017). Infants are superior in implicit crossmodal learning and use other learning mechanisms than adults. eLife 6:e28166. doi: 10.7554/eLife.28166
Roth, M. M., Dahmen, J. C., Muir, D. R., Imhof, F., Martini, F. J., and Hofer, S. B. (2016). Thalamic nuclei convey diverse contextual information to layer 1 of visual cortex. Nat. Neurosci. 19, 299–307. doi: 10.1038/nn.4197
Runyan, C. A., Piasini, E., Panzeri, S., and Harvey, C. D. (2017). Distinct timescales of population coding across cortex. Nature 548, 92–96. doi: 10.1038/nature23020
Russell, R. A. (2001). Survey of robotic applications for odor-sensing technology. Int. J. Robot. Res. 20, 144–162. doi: 10.1177/02783640122067318
Salinas, E., and Sejnowski, T. J. (2001). Correlated neuronal activity and the flow of neural information. Nat. Rev. Neurosci. 2:539. doi: 10.1038/35086012
Schanberg, S. M., and Field, T. M. (1987). Sensory deprivation stress and supplemental stimulation in the rat pup and preterm human neonate. Child Dev. 58, 1431–1447.
Scheier, C., Lewkowicz, D. J., and Shimojo, S. (2003). Sound induces perceptual reorganization of an ambiguous motion display in human infants. Dev. Sci. 6, 233–241. doi: 10.1111/1467-7687.00276
Schlaggar, B. L., and O’Leary, D. D. M. (1994). Early development of the somatotopic map and barrel patterning in rat somatosensory cortex. J. Comp. Neurol. 346, 80–96. doi: 10.1002/cne.903460106
Schroeder, C. E., and Lakatos, P. (2009). Low-frequency neuronal oscillations as instruments of sensory selection. Trends Neurosci. 32, 9–18. doi: 10.1016/j.tins.2008.09.012
Schroeder, C. E., Lindsley, R. W., Specht, C., Marcovici, A., Smiley, J. F., and Javitt, D. C. (2001). Somatosensory input to auditory association cortex in the macaque monkey. J. Neurophysiol. 85, 1322–1327. doi: 10.1152/jn.2001.85.3.1322
Sellers, K. K., Bennett, D. V., Hutt, A., Williams, J. H., and Frohlich, F. (2015). Awake vs. anesthetized: layer-specific sensory processing in visual cortex and functional connectivity between cortical areas. J. Neurophysiol. 113, 3798–3815. doi: 10.1152/jn.00923.2014
Senkowski, D., Schneider, T. R., Foxe, J. J., and Engel, A. K. (2008). Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401–409. doi: 10.1016/j.tins.2008.05.002
Sheppard, J. P., Raposo, D., and Churchland, A. K. (2013). Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans. J. Vis. 13:4. doi: 10.1167/13.6.4
Sherman, S. M. (2016). Thalamus plays a central role in ongoing cortical functioning. Nat. Neurosci. 19, 533–541. doi: 10.1038/nn.4269
Sieben, K., Bieler, M., Roder, B., and Hanganu-Opatz, I. L. (2015). Neonatal restriction of Tactile inputs leads to long-lasting impairments of cross-modal processing. PLoS Biol. 13:e1002304. doi: 10.1371/journal.pbio.1002304
Sieben, K., Roder, B., and Hanganu-Opatz, I. L. (2013). Oscillatory entrainment of primary somatosensory cortex encodes visual control of tactile processing. J. Neurosci. 33, 5736–5749. doi: 10.1523/JNEUROSCI.4432-12.2013
Siegel, F., Heimel, J. A., Peters, J., and Lohmann, C. (2012). Peripheral and central inputs shape network dynamics in the developing visual cortex in vivo. Curr. Biol. 22, 253–258. doi: 10.1016/j.cub.2011.12.026
Siegel, M., Buschman, T. J., and Miller, E. K. (2015). Cortical information flow during flexible sensorimotor decisions. Science 348, 1352–1355. doi: 10.1126/science.aab0551
Siegel, M., Donner, T. H., Oostenveld, R., Fries, P., and Engel, A. K. (2008). Neuronal synchronization along the Dorsal visual pathway reflects the focus of spatial attention. Neuron 60, 709–719. doi: 10.1016/j.neuron.2008.09.010
Siemann, J. K., Muller, C. L., Bamberger, G., Allison, J. D., Veenstra-VanderWeele, J., and Wallace, M. T. (2014). A novel behavioral paradigm to assess multisensory processing in mice. Front. Behav. Neurosci. 8:456. doi: 10.3389/fnbeh.2014.00456
Singer, W. (2009). Distributed processing and temporal codes in neuronal networks. Cogn. Neurodyn. 3, 189–196. doi: 10.1007/s11571-009-9087-z
Smith, P. H., Manning, K. A., and Uhlrich, D. J. (2010). Evaluation of inputs to rat primary auditory cortex from the suprageniculate nucleus and extrastriate visual cortex. J. Comp. Neurol. 518, 3679–3700. doi: 10.1002/cne.22411
Smith, S. L., and Trachtenberg, J. T. (2007). Experience-dependent binocular competition in the visual cortex begins at eye opening. Nat. Neurosci. 10:370. doi: 10.1038/nn1844
Song, Y. H., Kim, J. H., Jeong, H. W., Choi, I., Jeong, D., Kim, K., et al. (2017). A neural circuit for auditory dominance over visual perception. Neuron 93:940-954.e6.
Speechley, W. J., Hogsden, J. L., and Dringenberg, H. C. (2007). Continuous white noise exposure during and after auditory critical period differentially alters bidirectional thalamocortical plasticity in rat auditory cortex in vivo. Eur. J. Neurosci. 26, 2576–2584. doi: 10.1111/j.1460-9568.2007.05857.x
Stehberg, J., Dang, P. T., and Frostig, R. D. (2014). Unimodal primary sensory cortices are directly connected by long-range horizontal projections in the rat sensory cortex. Front. Neuroanat. 8:93. doi: 10.3389/fnana.2014.00093
Stein, B. E., Meredith, M. A., and Wallace, M. T. (1993). Chapter 8 The visually responsive neuron and beyond: multisensory integration in cat and monkey. Prog. Brain Res. 95, 79–90. doi: 10.1016/s0079-6123(08)60359-3
Stein, B. E., Scott Huneycutt, W., and Alex Meredith, M. (1988). Neurons and behavior: the same rules of multisensory integration apply. Brain Res. 448, 355–358. doi: 10.1016/0006-8993(88)91276-0
Stein, B. E., Wallace, M. W., Stanford, T. R., and Jiang, W. (2002). Book review: cortex governs multisensory integration in the midbrain. Neuroscientist 8, 306–314. doi: 10.1177/107385840200800406
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., and Harris, K. D. (2019). Spontaneous behaviors drive multidimensional, brainwide activity. Science 364:255. doi: 10.1126/science.aav7893
Tallon-Baudry, C., Bertrand, O., Hénaff, M. A., Isnard, J., and Fischer, C. (2004). Attention modulates Gamma-band Oscillations differently in the human lateral occipital cortex and fusiform gyrus. Cereb. Cortex 15, 654–662. doi: 10.1093/cercor/bhh167
Talsma, D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind. J. Integr. Neurosci. 9:19. doi: 10.3389/fnint.2015.00019
Talsma, D., Doty, T. J., and Woldorff, M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690. doi: 10.1093/cercor/bhk016
Talsma, D., Senkowski, D., Soto-Faraco, S., and Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14, 400–410. doi: 10.1016/j.tics.2010.06.008
Telley, L., Agirman, G., Prados, J., Amberg, N., Fievre, S., Oberst, P., et al. (2019). Temporal patterning of apical progenitors and their daughter neurons in the developing neocortex. Science 364:aav2522. doi: 10.1126/science.aav2522
Terreros, G., Jorratt, P., Aedo, C., Elgoyhen, A. B., and Delano, P. H. (2016). Selective attention to visual stimuli using auditory distractors is altered in Alpha-9 nicotinic receptor subunit knock-out mice. J. Neurosci. 36, 7198–7209. doi: 10.1523/JNEUROSCI.4031-15.2016
Teufel, C., and Nanay, B. (2017). How to (and how not to) think about top-down influences on visual perception. Conscious. Cogn. 47, 17–25. doi: 10.1016/j.concog.2016.05.008
Theyel, B. B., Llano, D. A., and Sherman, S. M. (2010). The corticothalamocortical circuit drives higher-order cortex in the mouse. Nat. Neurosci. 13, 84–88. doi: 10.1038/nn.2449
Thut, G., Nietzel, A., Brandt, S. A., and Pascual-Leone, A. (2006). α-Band Electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. J. Neurosci. 26, 9494–9502. doi: 10.1523/jneurosci.0875-06.2006
Tiesinga, P., Fellous, J. M., and Sejnowski, T. J. (2008). Regulation of spike timing in visual cortical circuits. Nat. Rev. Neurosci. 9, 97–107. doi: 10.1038/nrn2315
Toda, T., Homma, D., Tokuoka, H., Hayakawa, I., Sugimoto, Y., Ichinose, H., et al. (2013). Birth regulates the initiation of sensory map formation through serotonin signaling. Dev. Cell 27, 32–46. doi: 10.1016/j.devcel.2013.09.002
Tomita, H., Ohbayashi, M., Nakahara, K., Hasegawa, I., and Miyashita, Y. (1999). Top-down signal from prefrontal cortex in executive control of memory retrieval. Nature 401, 699–703. doi: 10.1038/44372
Treue, S., and Trujillo, J. C. M. (1999). Feature-based attention influences motion processing gain in macaque visual cortex. Nature 399, 575–579. doi: 10.1038/21176
Tuttle, R., Nakagawa, Y., Johnson, J. E., and O’Leary, D. D. (1999). Defects in thalamocortical axon pathfinding correlate with altered cell domains in Mash-1-deficient mice. Development 126, 1903–1916.
Tyll, S., Budinger, E., and Noesselt, T. (2011). Thalamic influences on multisensory integration. Commun. Integr. Biol. 4, 378–381. doi: 10.4161/cib.4.4.15222
Ueno, H., Suemitsu, S., Matsumoto, Y., and Okamoto, M. (2015). Sensory deprivation during early postnatal period alters the density of interneurons in the mouse prefrontal cortex. Neural Plast. 2015:753179. doi: 10.1155/2015/753179
Uziel, A., Romand, R., and Marot, M. (1981). Development of cochlear potentials in rats. Audiology 20, 89–100. doi: 10.3109/00206098109072687
Valeeva, G., Janackova, S., Nasretdinov, A., Rychkova, V., Makarov, R., Holmes, G. L., et al. (2019). Emergence of coordinated activity in the developing Entorhinal-Hippocampal network. Cereb. Cortex 29, 906–920. doi: 10.1093/cercor/bhy309
van Eden, C. G., and Uylings, H. B. M. (1985). Cytoarchitectonic development of the prefrontal cortex in the rat. J. Comp. Neurol. 241, 253–267. doi: 10.1002/cne.902410302
Vaneden, C. G., Lamme, V. A. F., and Uylings, H. B. M. (1992). Heterotopic cortical afferents to the medial prefrontal cortex in the rat - a combined retrograde and anterograde tracer study. Eur. J. Neurosci. 4, 77–97. doi: 10.1111/j.1460-9568.1992.tb00111.x
Vasconcelos, N., Pantoja, J., Belchior, H., Caixeta, F. V., Faber, J., Freire, M. A. M., et al. (2011). Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. PNAS 108, 15408–15413. doi: 10.1073/pnas.1102780108
Vertes, R. P. (2006). Interactions among the medial prefrontal cortex, hippocampus and midline thalamus in emotional and cognitive processing in the rat. Neuroscience 142, 1–20. doi: 10.1016/j.neuroscience.2006.06.027
Wallace, M. T., Meredith, M. A., and Stein, B. E. (1998). Multisensory integration in the superior colliculus of the alert cat. J. Neurophysiol. 80, 1006–1010. doi: 10.1152/jn.1998.80.2.1006
Wallace, M. T., and Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444. doi: 10.1523/jneurosci.17-07-02429.1997
Wang, H. C., and Bergles, D. E. (2015). Spontaneous activity in the developing auditory system. Cell Tissue Res. 361, 65–75. doi: 10.1007/s00441-014-2007-5
Wang, Q., and Burkhalter, A. (2007). Area map of mouse visual cortex. J. Comp. Neurol. 502, 339–357. doi: 10.1002/cne.21286
Webber, A., and Raz, Y. (2006). Axon guidance cues in auditory development. Anat. Rec. A Discov. Mol. Cell. Evol. Biol. 288, 390–396. doi: 10.1002/ar.a.20299
Weliky, M., Bosking, W. H., and Fitzpatrick, D. (1996). A systematic map of direction preference in primary visual cortex. Nature 379, 725–728. doi: 10.1038/379725a0
Welker, W. I. (1964). Analysis of sniffing of the albino rat. Behavior 22, 223–244. doi: 10.1163/156853964x00030
White, L. E., Coppola, D. M., and Fitzpatrick, D. (2001). The contribution of sensory experience to the maturation of orientation selectivity in ferret visual cortex. Nature 411, 1049–1052. doi: 10.1038/35082568
Wiese, E., Metta, G., and Wykowska, A. (2017). Robots as intentional agents: using neuroscientific methods to make robots appear more social. Front. Psychol. 8:1663. doi: 10.3389/fpsyg.2017.01663
Wimmer, R. D., Schmitt, L. I., Davidson, T. J., Nakajima, M., Deisseroth, K., and Halassa, M. M. (2015). Thalamic control of sensory selection in divided attention. Nature 526, 705–709. doi: 10.1038/nature15398
Winters, B. D., and Reid, J. M. (2010). A distributed cortical representation underlies crossmodal object recognition in rats. J. Neurosci. 30, 6253–6261. doi: 10.1523/JNEUROSCI.6073-09.2010
Wong, R. O. L., Meister, M., and Shatz, C. J. (1993). Transient period of correlated bursting activity during development of the mammalian retina. Neuron 11, 923–938. doi: 10.1016/0896-6273(93)90122-8
Yaka, R., Yinon, U., Rosner, M., and Wollberg, Z. (2000). Pathological and experimentally induced blindness induces auditory activity in the cat primary visual cortex. Exp. Brain Res. 132, 416–416. doi: 10.1007/s002210000392
Yang, J. W., An, S., Sun, J. J., Reyes-Puerta, V., Kindler, J., Berger, T., et al. (2013). Thalamic network oscillations synchronize ontogenetic columns in the newborn rat barrel cortex. Cereb. Cortex 23, 1299–1316. doi: 10.1093/cercor/bhs103
Yang, J. W., Hanganu-Opatz, I. L., Sun, J. J., and Luhmann, H. J. (2009). Three patterns of oscillatory activity differentially synchronize developing neocortical networks in vivo. J. Neurosci. 29, 9011–9025. doi: 10.1523/JNEUROSCI.5646-08.2009
Yang, J.-W., Reyes-Puerta, V., Kilb, W., and Luhmann, H. J. (2016). Spindle bursts in neonatal rat cereb cortex. Neural Plast. 2016, 3467832–3467832. doi: 10.1155/2016/3467832
Yau, J. M., DeAngelis, G. C., and Angelaki, D. E. (2015). Dissecting neural circuits for multisensory integration and crossmodal processing. Philos. Trans. R. Soc. B 370: 20140203. doi: 10.1098/rstb.2014.0203
Yu, L., Rowland, B. A., and Stein, B. E. (2010). Initiating the development of multisensory integration by manipulating sensory experience. J. Neurosci. 30, 4904–4913. doi: 10.1523/JNEUROSCI.5575-09.2010
Zhang, B., Shi, L. P., and Song, S. (2016). Creating more intelligent robots through brain-inspired computing, special supplement: brain-inspired intelligent robotics: the intersection of robotics and neuroscience sciences. Science 354, 4–9.
Zhang, S. Y., Xu, M., Chang, W. C., Ma, C. Y., Do, J. P. H., Jeong, D., et al. (2016). Organization of long-range inputs and outputs of frontal cortex for top-down control. Nat. Neurosci. 19, 1733–1742. doi: 10.1038/nn.4417
Zhang, L. I., Bao, S., and Merzenich, M. M. (2002). Disruption of primary auditory cortex by synchronous auditory inputs during a critical period. PNAS 99, 2309–2314. doi: 10.1073/pnas.261707398
Zhang, S., Xu, M., Kamigaki, T., Hoang, Do, J. P., Chang, W. C., et al. (2014). Selective attention. Long-range and local circuits for top-down modulation of visual cortex processing. Science 345, 660–665. doi: 10.1126/science.1254126
Keywords: cross-modal processing, primary sensory cortices, prefrontal cortex, top-down, bottom-up, development
Citation: Xu X, Hanganu-Opatz IL and Bieler M (2020) Cross-Talk of Low-Level Sensory and High-Level Cognitive Processing: Development, Mechanisms, and Relevance for Cross-Modal Abilities of the Brain. Front. Neurorobot. 14:7. doi: 10.3389/fnbot.2020.00007
Received: 31 August 2019; Accepted: 27 January 2020;
Published: 14 February 2020.
Edited by:
Alois C. Knoll, Technical University of Munich, GermanyReviewed by:
Benoît Girard, Centre National de la Recherche Scientifique (CNRS), FranceDenis Boire, Université du Québec à Trois-Rivières, Canada
Copyright © 2020 Xu, Hanganu-Opatz and Bieler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ileana L. Hanganu-Opatz, aGFuZ29wQHptbmgudW5pLWhhbWJ1cmcuZGU=; Malte Bieler, bWFsdGViQG1lZGlzaW4udWlvLm5v