The relationship between electrophysiological and hemodynamic measures of neural activity varies across picture naming tasks: A multimodal magnetoencephalography-functional magnetic resonance imaging study

Different neuroimaging methods can yield different views of task-dependent neural engagement. Studies examining the relationship between electromagnetic and hemodynamic measures have revealed correlated patterns across brain regions but the role of the applied stimulation or experimental tasks in these correlation patterns is still poorly understood. Here, we evaluated the across-tasks variability of MEG-fMRI relationship using data recorded during three distinct naming tasks (naming objects and actions from action images, and objects from object images), from the same set of participants. Our results demonstrate that the MEG-fMRI correlation pattern varies according to the performed task, and that this variability shows distinct spectral profiles across brain regions. Notably, analysis of the MEG data alone did not reveal modulations across the examined tasks in the time-frequency windows emerging from the MEG-fMRI correlation analysis. Our results suggest that the electromagnetic-hemodynamic correlation could serve as a more sensitive proxy for task-dependent neural engagement in cognitive tasks than isolated within-modality measures.

Introduction

Functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) are widely used non-invasive neuroimaging methods that both have their own strengths. FMRI measures hemodynamic modulations resulting from multiple neural and vascular phenomena, and yields an accurate three-dimensional map of brain activity with a good spatial resolution. However, its temporal precision is modest due to the sluggishness of the hemodynamic response. MEG measures the magnetic field elicited by electric activity, and possesses a temporal resolution in the millisecond scale. The distribution of cortical activity yielded by MEG is relatively smooth.

Thus, depending on the importance of the spatial and temporal aspects of brain activity to the research question at hand, either method may be the preferred option. It has also been proposed that combining MEG and fMRI would allow one to obtain accurate spatiotemporal maps of brain activity (Dale et al., 2000; Henson et al., 2010). Such approaches have, for example, utilized fMRI to spatially constrain the MEG source-level estimates, either explicitly (Matchin et al., 2019) or in a probabilistic manner (Cottereau et al., 2015; Wang and Holland, 2022). Another principle that has been applied to combine electrophysiological and hemodynamic signals is to use the MEG or electroencephalography (EEG) based individual measures of neural activity to model the fMRI response instead of common regressors across subjects, leading to increased statistical power (Renvall et al., 2012a; Iannaccone et al., 2015). In this effort, various kinds of computational analyses have been applied to obtain more comprehensive spatiotemporal accounts than can be afforded by MEG or fMRI data alone. In some instances, machine-learning based classification analyses have been conducted separately for MEG and fMRI data to obtain maximally accurate temporal and spatial accounts of neural phenomena (Brandman and Peelen, 2017). Computational models have been applied to identify electrophysiological correlates of behavioral processes which, in turn, have been used to model the trial-level variability within fMRI signals (Pisauro et al., 2017). Some studies have further utilized representational similarity analyses across MEG and fMRI data to accomplish spatially and temporally detailed characterization of neuronal activity (Cichy et al., 2014, 2016; Leonardelli and Fairhall, 2022). Generative models have also been utilized to capture the state transitions in both fMRI and MEG resting-state networks (Jiang et al., 2022). In a clinical setting, fusion of distinct neuroimaging measures, such as MEG and fMRI, is increasingly being used to improve the ability to distinguish between patient groups and control participants (Calhoun and Sui, 2016). Together, these reports demonstrate the versatile ways that MEG and fMRI can be merged to obtain spatiotemporally detailed accounts of neural-level processing.

In order to use electrophysiological and hemodynamic measures together in a principled manner, it is important to understand the relationship between the different types of measures. Numerous neuroimaging studies have therefore attempted to deepen this understanding. Initially, the emphasis was on how electrophysiological and hemodynamic techniques would allow the identification and localization of neural responses to the same types of stimuli (Sanders et al., 1996; Stippich et al., 1998). Subsequently, the focus has been more on identifying electrophysiological phenomena that correlate with the blood-oxygen-level dependent (BOLD) fMRI signal. In general, such studies have revealed robust and spectrally systematic correlation patterns between neural and hemodynamic activity (Logothetis et al., 2001; Mukamel et al., 2005; Scheeringa et al., 2011). However, it has also been shown that the correlation between electrophysiological measures and the BOLD signal varies across brain regions (Conner et al., 2011; Kujala et al., 2014). Moreover, comparisons between large-scale networks derived from MEG and fMRI have indicated a complex frequency-specific relationship between fMRI and the electrophysiological connectivity (Hipp and Siegel, 2015; Liljeström et al., 2015b). Furthermore, studies examining the MEG and fMRI signals in identical experimental settings from the same subjects have revealed systematic functional differences between the electrophysiological and BOLD responses (Liljeström et al., 2009; Vartiainen et al., 2011). Accordingly, it is commonly accepted that when integrating the temporally and spatially accurate views of neural processing from MEG and fMRI, it is crucial to consider the complex nature of the origins of hemodynamic fluctuations (Logothetis, 2008; Ekstrom, 2010; Lauritzen et al., 2012; Whitman et al., 2013).

One aspect that has received less attention in combining MEG and fMRI measures both for obtaining detailed spatiotemporal accounts as well as investigating neurovascular coupling has been the role of the applied stimulation or experimental tasks themselves. Naturally, a broad range of stimuli from different sensory modalities as well as various kinds of cognitive experiments have been applied. However, the main goal in those explorations has been to induce detectable signals in different neural systems across the cortex and to develop approaches that maximize the association between the two signal types (Lankinen et al., 2018), not to examine how the different stimuli and tasks might influence the joint modulation of electrophysiological and hemodynamic signals. Yet, it has been shown that the local neural and hemodynamic signals can be partially decoupled (O’Herron et al., 2016), and that the relationship between electrophysiological and hemodynamic signals depends on the correlation between the local inputs (Butler et al., 2017), effects that could cause variability in the MEG-fMRI correlations across stimuli and tasks. In the present study, we sought to explicitly utilize the inherently complex relationship between BOLD fluctuations and modulations of electrophysiological brain activity as well as the possible task-induced variability in this relationship to track and dissociate the neural engagement of different brain regions across distinct cognitive tasks. We asked whether any differences in neural processing related to distinct picture naming tasks could be highlighted through MEG-fMRI fusion as compared to isolated within-modality measures. Specifically, we investigated the variability of MEG-fMRI correlation patterns across three naming tasks (naming objects and actions from action images, and objects from object images) using a dataset where MEG and fMRI data were recorded from the same participants in identical experiments (Liljeström et al., 2009). The correlation patterns were obtained by first computing the MEG-fMRI correlation separately for each brain region, time window and frequency band, and by then applying variance minimizing hierarchical clustering to find clusters of similarly correlated brain areas. The approach allows the grouping of both task-invariant and task-dependent correlation patterns across brain regions regardless of their spatial adjacency. We predicted that our approach would reveal both types of correlation patterns and, critically, facilitate identification of neural engagement that could not be detected using one imaging modality alone.

Materials and methods

Subjects

Magnetoencephalography and fMRI data were collected from 10 healthy (nine right-handed, one ambidextrous), native Finnish-speaking subjects (four females, six males; ages 20–33 years). Informed consent was obtained from all subjects, in agreement with a prior approval of the Local Ethics Committee (Hospital District of Helsinki and Uusimaa). The subjects did not report any neurological disorders, and all had normal or corrected-to-normal vision. All methods were conducted in accordance with the guidelines of the Finnish National Board on Research Integrity.

Experimental design

The task was to silently name pictures of objects or actions presented as simple black line art on a gray background. There were two categories of drawings. In the first category, an action performed with an object was depicted, whereas in the second category, a single object was shown. To achieve the same visual complexity as in action images the object images were constructed from the action images by dissolving the action figures into non-meaningful lines in the background. The experiment consisted of three different cognitive tasks: Object naming from object images (100 trials), action naming from action images (100 trials), and object naming from action images (100 trials). The experiment had a blocked design with 10 stimuli of the same task presented within each 30-s block. Each image was shown for 300 ms at 1.8–4.2 s intervals. Each block started with an instruction indicating the task for the block. The task blocks were separated by 21-s rest blocks. The experiment was divided into two runs, with different sets of stimuli in the two runs (150 images per run, 50 per task). The experimental design was identical in MEG and fMRI leading to two matched runs per subject. The order of the three naming conditions was randomized in both runs and silent naming was used to avoid muscular artifacts. A complete description of the experiment can be found in Liljeström et al. (2009). The design permits identification of effects that are related to the naming task (comparing action naming to both object naming conditions) and to the picture type (comparing object-only images to both action image conditions). Behaviorally, (overt) object naming from action images leads to longer reaction times than for naming objects from object images or actions from action images (Liljeström et al., 2015a), indicating that increased effort or additional processing is required when naming objects from action images compared to the two other tasks. It is therefore of interest also to compare the object naming from action image condition to the other tasks.

Functional magnetic resonance imaging data collection

The MRI data were collected at the Advanced Magnetic Imaging Centre (Aalto University) with a Signa VH/i 3.0 T MRI scanner (GE Healthcare, Chalfont St Giles, UK). Anatomical MRIs were acquired using a T1-weighted 3D spoiled gradient-echo sequence. Functional MRI data were collected using a single-shot gradient-echo planar imaging sequence (TR 3 s, TE = 32 ms, FA = 90, slice thickness 3 mm, in-plane resolution either 3 mm × 3 mm, or 3.4 mm × 3.4 mm). The first five functional volumes were discarded from the analysis.

Magnetoencephalography data collection

Magnetoencephalography data recordings were conducted using a 306-channel whole-head device (Elekta Oy, Helsinki, Finland) in a magnetically shielded room. The data were bandpass filtered to 0.03–200 Hz and sampled at 600 Hz. The temporal extension of the Signal Space Separation method (Taulu and Simola, 2006) was applied in order to suppress contributions from external artifacts. Eye movements were monitored with electro-oculogram (EOG).

Functional magnetic resonance imaging and magnetoencephalography data analysis

The overview of the analysis pipeline including key formulae for the conducted computations is presented in Figure 1. First, to facilitate the across-subjects evaluation of MEG-fMRI correlation, the data of each subject were transformed to an average brain via a surface-based transformation (Fischl et al., 1999) using Freesurfer 5.3 (Fischl, 2012). Before the transformation, the individual fMRI data were realigned to the first volume and susceptibility artifacts caused by movements were corrected for using SPM8 (Wellcome Department of Cognitive Neurology, London, UK). The mean image of the functional series was used for co-registering the fMRI data with the individual anatomical images. For each vertex in the average brain, the fMRI values of a spatially matching voxels were then taken to represent the fMRI activity at the cortical surface level.

FIGURE 1

Figure 1. An outline of MEG and fMRI analysis pipelines, displaying the most important steps and their order. Gray boxes show essential normalizations that aim to equalize the measures obtained with the two modalities.

The vertex-level data were averaged within 188 parcels covering the entire cortical surface (see e.g., Figure 3). This parcellation was based on the automatic anatomic parcellation of the human cortical gyri and sulci consisting of 144 parcels (Destrieux et al., 2010) that was subsequently computationally modified to form a parcellation that would be more suitable for the analysis of MEG data. Specifically, the parcellation scheme was obtained by applying a PCA algorithm implemented in MNE-python (Gramfort et al., 2013) for splitting parcels defined by the automatic anatomical labeling scheme of the cortical surface (Destrieux et al., 2010). The splitting yields parcels that are relatively symmetrical and small enough to be relatively homogeneous with respect to local activations. Notably, the splitting was based solely on the anatomical information without utilizing functional data, leading to a parcellation that is less optimized (Thirion et al., 2014) but more generalizable to multiple datasets. While the exclusively anatomical parcellation obtained via splitting the original parcels using spatial PCA does not ensure an exact alignment between MEG and fMRI responses and the parcels, it allows better separation of responses that are spatially distinct than when using the original Destrieux atlas. From the entire parcellation consisting of 188 parcels, regions that are prone to artifacts or signal loss in either of the imaging methods (anterior parts of the frontal lobe, deepest parts of the medial surface, and inferior parts of the temporal lobe) were omitted. The set of parcels used in the final analysis consisted of 70 parcels per hemisphere.

In the fMRI analysis the goal was to determine, for each fMRI run and experimental condition, the BOLD signal change with respect to rest within each parcel. This was accomplished by first high-pass filtering the parcel-level representation of the fMRI data in SPM8 with a cut-off frequency of 1/510 Hz. Baseline effects were removed using a rest block (6 volumes) that preceded each stimulus block (11 volumes), thus removing slow drifts taking place during the scanning runs. For each fMRI block and parcel, the data were averaged across the collected 11 volumes. The data were then normalized by subtracting the mean activity across all blocks and tasks from the block and task specific values, and by dividing these values by the standard deviation of the whole run’s data. The normalization was done separately for each subject to remove inter-subject differences in signal scales and means. Subsequently, within each run, blocks of each task were averaged per participant. For the correlation analysis, we thus obtained a total of 20 fMRI values per task (10 participants, 2 runs).

Magnetoencephalography data estimates were obtained for the same parcels in six different frequency bands: Theta (4–7 Hz), alpha (8–13 Hz), low beta (15–21 Hz), high beta (23–29 Hz), low gamma (36–46 Hz), and high gamma (54–90 Hz), from 100 to 800 ms with respect to stimulus presentation. The gamma-band analyses were conducted in two separate bands to avoid including 50 Hz line noise into the estimates. The estimation was done using event-related Dynamic Imaging of Coherent Sources (Laaksonen et al., 2008), a beamforming technique in the time-frequency domain. Here, only data from the 204 gradiometers were used. In the estimation, a surface-based grid consisting of 5,122 points was first created in the average brain with MNE (Gramfort et al., 2014) and transformed to each individual’s anatomy using Freesurfer 5.3 (Fischl, 2012). Brain activity estimates for each task and block were then computed for each grid point in the six different frequency bands, in 22 partially overlapping 200-ms time-windows (33-ms time difference between two successive time-windows). The 200-ms window length was chosen as it was the shortest length that allowed the accurate estimation of data covariance and thus brain activity given the signal-to-noise ratio (SNR) and number of trials across experimental tasks within the present dataset (see, e.g., Brookes et al., 2008). This window length is likely to be sufficiently short for exploring the sustained neural phenomena in higher-order cortical regions but could be sub-optimal for determining the temporally intricate early processes within the visual hierarchy. A baseline value was computed from the prestimulus interval −200 to 0 ms, separately for each block and grid point. Trials in which the amplitude of either the vertical or the horizontal EOG exceeded 150 μV were rejected. The parcel-level values were obtained by calculating, in each grid point, the difference between the post-stimulus values and the corresponding baseline values (divided by the same baseline values) and computing the average across all these baseline relative changes within each parcel. The parcel values were normalized separately for each subject, run and frequency band. This was done similarly as for the fMRI data, by subtracting the average activity across all blocks and tasks from the block and task specific values, and by dividing these values by the standard deviation of the data from the entire run. The run-level data were then obtained by averaging the block-specific data within each run. Similarly to the fMRI data, we thus obtained a total of 20 MEG values per task (10 participants, 2 runs) for the correlation analysis.

Correlation analysis and clustering

We computed a vector of MEG-fMRI correlation estimates for each parcel using Spearman’s rank correlation (see Figure 2). Within a parcel, separate correlations were computed for all tasks, time intervals and frequency bands (3 tasks, 22 time-windows, 6 frequency bands: in total 396 MEG-fMRI correlation estimates per parcel). Each of these estimates was computed based on 20 MEG and 20 fMRI observations (10 subjects and 2 runs). We applied an agglomerative (merging) hierarchical clustering algorithm on our Spearman’s rho value vectors to find clusters of similarly correlated regions. For this, we used the Ward minimum variance method (Ward, 1963) that aims to minimize the within-cluster variance, leading to a clustering where correlation patterns inside a cluster are as similar as possible [function linkage(…,“ward”) in Matlab]. Information about hemispheres was not passed to the clustering algorithm. Ward’s method measures Euclidean distances between cluster centroids during its merging steps. The clustering algorithm produces a hierarchical cluster tree structure that describes the merging process. The leaves of the tree can be reordered without changing the structure itself. The optimal leaf order is such that the similarities of adjacent leaves are maximized [function optimalleaforder() in Matlab]. The final clustering allows for visual comparison of the correlation patterns across the three tasks and different frequency bands.

FIGURE 2

Figure 2. Matrix of correlations between MEG and fMRI for the three experimental conditions (separated with thick white vertical lines). Each condition-related submatrix is divided into six frequency bands: Theta, alpha, low beta, high beta, low gamma, and high gamma, from left to right (columns separated by thin vertical gray lines). Each frequency band consists of a sequence of 22 time points (sub-columns). All 140 brain regions (70 per hemisphere) are displayed on the y-axis, ordered with respect to the optimal leaf order of a cluster tree. This leads to a solution where distances between similarly behaving brain regions are minimized. The brain regions (rows) are divided into 17 clusters (C1–C17, separated by horizontal thin gray lines; see Figure 3 for visualization of the areas on MRI). The clustering is the same for all three conditions. The color indicates the MEG-fMRI correlation strength (–1…+1), see scale on the right.

FIGURE 3

Figure 3. Clustering of brain regions. Level 17 of a clustering tree (used in the analyses). The deep medial and anterior frontal areas plotted in light gray were omitted from the analysis. Clusters are ordered according to the optimal leaf order and marked with labels C1–C17 (cf. Figure 2).

To evaluate the possible differences in correlations across the tasks, we estimated the 99% confidence limits for each task across the identified clusters, separately for left and right-hemisphere parcels, using bootstrapping (Efron, 1979). The bootstrapping was conducted by re-sampling the data 10,000 times, by computing the new MEG-fMRI correlation values for each sample, and by estimating the 99% confidence limits for each task from the obtained distribution. In the re-sampling, 80% of the data were randomly selected at each round. In this evaluation, we only considered those clusters and frequency bands in which at least one of the three tasks showed significant MEG-fMRI correlation (p < 0.05, Bonferroni-corrected over time points).

To compare a joint analysis approach and a more conventional approach utilizing a single brain imaging method alone, we also evaluated the differences in the MEG activity patterns between the tasks with paired t-tests (p < 0.05, Bonferroni-corrected over time points) for the identified clusters. This analysis was performed with the same temporal and spectral resolution as the MEG-fMRI correlation analysis and was, thus, only applicable to MEG; fMRI lacks the temporal resolution that would be needed for comparison of fMRI activity and MEG-fMRI correlation modulations. The comparison was therefore restricted to MEG activity and MEG-fMRI correlation patterns. Potential differences in the temporal-spectral aspects of the findings between the two approaches would reveal unique results that can be achieved only with one of the approaches, but not both.

Results

Clustering of correlation patterns

For clustering purposes, a matrix was constructed (see Figure 2), where each row lists the MEG-fMRI correlation values across the different frequency bands, time points and tasks. The clustering algorithm enables identification of clusters in which all three tasks behave similarly, but also clusters in which the tasks behave differently. In Figure 2, the rows are reordered according to a full cluster tree so that similar rows are close to each other. The ordering reveals salient MEG-fMRI correlation patterns, with consistent negative and positive correlation patterns across brain regions. The selected clustering consists of 17 clusters (Figure 3), chosen based on an appropriate level of spatial separation across parcels. With a smaller number of clusters, functionally distinct brain regions remain in larger shared clusters, whereas with a larger number of clusters single parcels start to form clusters by themselves. Accordingly, with a smaller set of clusters, regions with functionally distinct activity profiles would be merged together, whereas with a larger set of clusters individual parcels with very similar activity profiles would be segregated into distinct clusters. In general, the clusters were spatially concentrated, indicating that close-by regions show more similar MEG-fMRI correlation patterns than regions that are further apart. The clustering (Figure 3) agreed well with the known functional division of cortical processing related to picture naming, revealing, e.g., components representing both lower (C8) and higher-order (C16) visual, speech related motor/premotor (C5 and C7), and perisylvian language related processing (C9). In particular, lower-order regions involved in the basic visual processing formed clusters (C8, C11, C16, and C17) that did not include any higher-order cortical areas, whereas the clusters containing higher-order regions generally represented distinct neural functions associated with different cortical lobes and also with more fine-grained differences (e.g., separation of inferior vs. superior frontal cortices and lateral vs. medial cortical structures). Many of the identified clusters (C1, C4, C8, C9, C11, C12, C15, and C17) showed marked symmetry across the hemispheres, but temporal, central and inferior frontal cortical areas (e.g., C2, C5–C7, and C13–C14) critically involved in picture naming tended form clusters exclusively within individual hemispheres.

Magnetoencephalography-functional magnetic resonance imaging correlation differences between tasks

For the clusters, we determined significant differences in MEG-fMRI correlation spectra between experimental conditions, across multiple frequency bands and time-windows (see Table 1 and Figure 4). We focused on identifying effects where one of the conditions differed from the other two conditions: (i) naming actions differed from both object naming conditions (different tasks; Figure 4, rectangles with solid orange line), (ii) naming objects from object pictures differed from naming objects or actions from action pictures (different images; Figure 4, rectangles with dotted black line), and (iii) naming objects from action pictures differed from both naming objects from object pictures and naming actions from action pictures (different reaction times; Figure 4, rectangles with solid gray line). Correlations were examined separately for parcels within each hemisphere.

TABLE 1

Table 1. Significant effects detected with the given clusters, frequency bands, and significant time intervals.

FIGURE 4

Figure 4. Magnetoencephalography-Functional magnetic resonance imaging correlation patterns divided into clusters (row labels) and hemispheres (left and right panels). The three rows in each cluster show correlation between fMRI and MEG for the three experimental conditions: from top to bottom, object naming from object images, action naming from action images and object naming from action images, over time in the different frequency bands (column labels). Significant correlations (p = 0.05, Bonferroni-corrected over the 22 time points) are marked as thicker parts of stripes. Rectangles indicate areas where the 99% confidence intervals of one condition do not overlap those of the other two conditions. A salient difference between naming tasks (naming actions vs. objects) is denoted by an orange rectangle, a difference between two picture types (action vs. object stimulus) is indicated by a dotted black rectangle, and a difference specific to naming objects from action images vs. the other two tasks with a gray rectangle. A rectangle is shown only when there is also a significant MEG–fMRI correlation inside the rectangle. Clusters C3, C5-C7, and C10 have parcels only in the left hemisphere (blank gray bars in the right-hemisphere).

Modulations of MEG-fMRI correlation across-tasks were detected predominantly in the left hemisphere. Different picture types elicited distinct correlation patterns in the occipital and parietal cortex, within the alpha and gamma frequency bands (left-hemisphere clusters C1, C3, and C13; see Figure 5A and Table 1). Between different naming tasks, correlations differed along the central sulcus and the posterior temporo-parietal cortex, mainly in the left hemisphere (left-hemisphere clusters C5, C6, and C10 and right-hemisphere cluster C2), particularly in the gamma-range. Distinct correlation patterns for the condition in which the participants named objects from action images as compared to the other two categories were observed exclusively in the left hemisphere and included brain regions within the posterior temporo-parietal cortex (cluster C10) as well as within the occipital cortex (clusters C11 and C17), with contributions from the theta band as well as low and high gamma-bands.

FIGURE 5

Figure 5. Magnetoencephalography-functional magnetic resonance imaging correlation as a function of time. For each cluster, the top row shows the correlation spectra for all tasks (naming object from action pictures in gray; naming actions in orange; and naming objects from object pictures in dotted black), and the bottom row the 99% confidence intervals for the three tasks (correspondingly gray, orange and a striped black pattern). In the correlation spectra, the colored squares indicate time instances at which the correlation is significant (p = 0.05, Bonferroni-corrected over time). (A) Task-dependent instances: One task shows significant correlation and differs from the other two tasks (non-overlapping confidence bounds) at a given time. White areas between the confidence intervals of experimental conditions indicate time instances of significantly different MEG-fMRI correlation between two or more conditions (p = 0.01, uncorrected). (B) Task-invariant instances: Clusters 16 and 17 suggest consistent negative correlation between MEG and fMRI at lower frequencies, among all experimental conditions, in the occipital cortex.

Task-invariant magnetoencephalography-functional magnetic resonance imaging correlation patterns

Figures 5B Shows the correlation patterns for two clusters within the occipital cortex (clusters C16 and C17). Parcels in the left-hemisphere cluster C17, covering the middle occipital cortex, and those in the right-hemisphere cluster C16, covering the medial and lateral parts of the occipital cortex, showed a significant negative MEG-fMRI correlation at low frequencies, but not in the gamma-range.

Magnetoencephalography activation vs. magnetoencephalography-functional magnetic resonance imaging correlation

Across the 17 identified clusters, the time-frequency windows in which MEG-fMRI correlation showed task-dependent modulation were highly distinct from the time-frequency windows in which MEG activity was modulated (Figure 6). Modulation of correlation was observed mainly in early time-windows (<500 ms), whereas modulations of activity (with band-limited power as measure) were exclusively detected more than 500 ms after stimulus onset. In the frequency domain, the MEG activity modulations were concentrated to the theta, alpha and (low and high) beta bands, whereas the MEG-fMRI correlation effects also showed a prominent contribution of gamma-band neural activity. No significant MEG-fMRI correlation effects were detected in the high beta band. Significant differences in activation were detected between object naming from object vs. action images, as well as for object naming from object images vs. action naming from action images; however, no differences were observed between object vs. action naming from action images (Figure 7). These effects were particularly prominent within the left hemisphere, predominantly in clusters with parcels in the parietal lobe. No significant effects of MEG signal changes were detected between object and action naming from identical images, in contrast to the MEG-fMRI correlation analysis which identified several left-hemisphere clusters in which action naming differed from the other two conditions (C2, C5, C10, and C13, Figure 5).

FIGURE 6

Figure 6. Temporo-spectral uniqueness and overlap in modulation of rhythmic activity and MEG-fMRI correlation. Timing with respect to picture presentation is plotted on the x-axis, and the different frequency bands on the y-axis. Time-frequency windows that showed differences between the conditions only for MEG band-limited power (light gray), only for MEG-fMRI correlation (dark gray) or both (black). Values averaged across all contrasts.

FIGURE 7

Figure 7. Significant results in the MEG activation analysis for each cluster. For each cluster the number of significant time-frequency bins are indicated as the percentage of all possible time-frequency bins (in total 132 bins from 6 frequency bands and 22 time-windows). The bars are color-coded according to the lobe to which the majority of the parcels belong to. The clusters are ordered according to the total percentage of significant time-frequency bins in all tasks (left and right- hemispheres separately). Note that there were no significant differences between neural activity during Object naming from action images and action naming, whereas for the other two contrasts where the stimulus contents were different multiple clusters showed significant differences.

Discussion

We have shown that correlation between MEG and fMRI contains information that distinguishes between the three naming tasks. This finding aligns with observations that have demonstrated trial and stimulus dependent variability in the relationship between electrophysiological and hemodynamic activity within the visual cortex (O’Herron et al., 2016; Butler et al., 2017). Furthermore, our results demonstrate that the time-frequency windows in which the MEG-fMRI correlation patterns differ between the tasks are distinct from the windows showing task effects in a separate MEG-based analysis of modulation of neural activity. Interestingly, the differences in the correlation patterns between tasks were typically observed in markedly transient time-windows, highlighting the dynamic nature of the neural phenomena dissociating the different picture naming conditions also at the level of MEG-fMRI correlations. Notably, such correlation differences were not specific to any frequency bands but extended to a wide range of distinct oscillations (theta, alpha, beta, and gamma). On the other hand, task-invariant correlations especially in the theta- and alpha-bands tended to be more sustained, attesting to the distinct nature of task-dependent vs. task-invariant correlation patterns. From amongst the 17 identified clusters, nine showed significant differences between the three experimental conditions whereas no differences were observed in the other clusters covering, in particular, more anterior lateral frontal areas and primary visual cortices. Significant differences were observed for all contrasts in the parietal cortex, with more superior effects for different images and more inferior effects for different tasks and conditions with different reaction times. Differences in the MEG-fMRI correlation patterns were also observed for different images in the anterior medial frontal cortex and for different tasks in the post- and precentral gyri. Notably, the involvement of the parietal cortex was detected also in the analyses focusing on the MEG and fMRI activity, whereas the role of the anterior medial frontal cortex and the post- and precentral gyri in dissociating the different naming conditions was not observed in these studies (Liljeström et al., 2008, 2009). Our results thus illustrate that the multimodal correlations yield novel information about the task-dependent neural engagement that cannot be detected using one imaging method alone.

Detection of neural engagement using multiple neuroimaging methods

Task-dependent processing in neural circuits is a complex phenomenon that is supported by a wide range of mechanisms involving, e.g., electric, metabolic, and neurotransmitter activity (Singh, 2012). Measuring any of these processes yields one particular view of the full activity of the circuit. As it is not feasible to simultaneously record all possible processes related to the engagement of a circuit, its full activity remains a variable that may be estimated using specific proxies. As individual proxies are noisy and give incomplete information, it may not be possible to accurately estimate the full brain activity in a region. Thus, the observed activation patterns determined by an individual proxy may not reveal any observable brain activity even if the neural circuit, in reality, participates in task-dependent processing. The same holds when the goal is to determine differences between levels of neural engagement between different experimental conditions.

It has been proposed that the complexity of the human brain coupled with the incomplete measurements make multimodal data fusion critical for identifying detailed, individual-level properties of brain anatomy and function (Calhoun and Sui, 2016). Multimodal data-fusion based approaches have proven particularly useful for combining genetic mapping with other measures in the study of brain disorders (Purcell et al., 2009; Pearlson et al., 2015) as well as for evaluating the variability of brain anatomy and function in healthy subjects (Hardoon et al., 2009; Le Floch et al., 2012; Renvall et al., 2012b; Salmela et al., 2016) and predicting the subjects’ age (Engemann et al., 2020). So far, fusion of different neuroimaging data-types has been applied for identifying (in individual brain regions), e.g., the neural underpinnings of the BOLD response (Scheeringa et al., 2011; Kujala et al., 2014), also at the laminar level (Scheeringa et al., 2016; Warbrick, 2022), the effects of anatomical properties on functional data (Sepulcre et al., 2009; Schwarzkopf et al., 2012), or the effects of GABAergic inhibition on fMRI and MEG responses (Muthukumaraswamy et al., 2009; Kujala et al., 2015). While it has been proposed that by combining the temporally/spectrally and spatially sensitive measures of neural engagement provided by MEG and fMRI one could obtain a spatiotemporally accurate picture of brain activity (Dale et al., 2000), such data fusion has rarely been applied. Moreover, this type of combination has typically been used only in the primary sensory and motor neural systems (Schulz et al., 2004; Whittingstall et al., 2007; Stevenson et al., 2012; Renvall et al., 2012a; Cichy et al., 2014). Recently, similarity-based fusion methods combining MEG and fMRI have proven useful in learning relationships between visual objects and how they are represented within the visual system (Cichy et al., 2016) as well as within the semantic system (Leonardelli and Fairhall, 2022). In cognitive tasks, the improvement of SNR through group-level analysis (increased amount of data) may be limited by notable inter-subject variability, leading to a failure to detect the true engagement of neural circuits, even when multiple proxies are combined.

In the present study, we aimed to develop and apply a data-fusion based approach that would explicitly utilize the inter-subject and inter-block variability in combining different measurements (MEG and fMRI) to build a more sensitive and accurate picture of the neural engagement. Specifically, we used the correlation between MEG estimates of induced activity in different time-frequency windows and BOLD-fMRI estimates of hemodynamic activity to determine the neural circuits that are engaged in a distinct manner in three picture naming tasks. The MEG and fMRI proxies of neural activity can occasionally show salient negative or positive correlation when the brain activity is strong enough to be detected. In areas where one imaging method yields only noise and the other a good signal, task-wise correlations cannot be significant. To detect activity in a neural circuit, our approach requires that there is a causal connection between the engagement of the circuit and the two proxies (MEG and fMRI). Notably, unlike in typical neuroimaging studies, the sensitivity of the approach to detect neural engagement is in fact increased if the subjects or the blocks show considerable variability, given that the assumption of causality is met. In general, our approach as well as other approaches that profit from such variability are likely to be beneficial in cases where the SNR is low and where there is large individual variance in elicited neural processes. Hence, this type of approaches should prove useful in detecting neural engagement particularly in cognitive tasks.

Multimodal correlation as a spatially, temporally and spectrally unique view on neural engagement during picture naming

In the present study, we applied the developed MEG-fMRI correlation based method to a picture naming data set that had been previously analyzed separately using traditional MEG (evoked responses) and fMRI group-level statistical approaches for identification of neural activity related to different naming tasks (action vs. object naming) (Liljeström et al., 2009) as well as identification of task-relevant functional networks (Liljeström et al., 2015a). Several studies have shown a negative correlation between MEG and fMRI at lower alpha and beta frequencies, and a positive correlation within the gamma frequency range, especially in low-level sensory cortices (Logothetis et al., 2001; Mukamel et al., 2005; Scheeringa et al., 2011). In higher-level cortical regions and in cognitive tasks this relationship is more variable (Conner et al., 2011; Kujala et al., 2014). Moreover, analysis of functional networks has indicated a complex frequency-dependent relationship between MEG- and fMRI-derived networks that varies across-tasks (Liljeström et al., 2015b). In the present study, we observed task-invariant negative correlations between MEG and fMRI within the alpha and beta frequency bands in occipital and parietal regions, in line with previous studies (Logothetis et al., 2001; Scheeringa et al., 2011).

Our main goal was, however, to utilize the variability in the relationship between MEG and fMRI and identify clusters that manifested a task-varying relationship in MEG and fMRI correlation. This correlation-based approach revealed significant differences between the conditions in which the activation based analysis had not done so. Within the left parieto-temporal junction, along the central sulcus, and the inferior frontal cortex, the correlation pattern was different between the action naming condition and the two object naming conditions. In contrast, MEG activation analysis either with induced responses in the present study, or previously with evoked responses (Liljeström et al., 2009), did not reveal significant differences between action and object naming from identical images. These effects demonstrate that the correlation-based analysis can reveal neural engagement in functionally relevant circuits that are not detected in conventional activation-based analyses.

The most notable new insights revealed by the present approach were the spectral and temporal patterns of electrophysiological activity. For example, the correlation patterns differed in the parieto-occipital cortex for the conditions where the stimulus content was different. In the present analysis of the modulation of induced activity, effects were detected in late time-windows (>500 ms), whereas the correlation patterns revealed differences primarily in notably earlier intervals (200–400 ms). These findings suggest that the modulation of alpha/beta activity is distinct for different stimulus contents, a finding that could not be inferred from traditional analysis of MEG activation; the results also demonstrate that these early differences are linked with the BOLD activity that is measured in those cortical regions. Secondly, the correlation-based analysis revealed, in contrast to analysis of MEG induced activity, prominent effects in the gamma-band. This suggest that the present multimodal analysis may help reveal the role of high-frequency neural activity in cognitive processing that is often difficult to detect with non-invasive techniques.

Detection of cortical activity using clustering of magnetoencephalography-functional magnetic resonance imaging correlation patterns

In the present study, we computed the correlation between individual-level, run-wise MEG and fMRI recordings of the same experimental conditions from the same subjects. The goal was to develop an approach that would utilize the correlation between the two distinct proxies (MEG and fMRI) of brain activity to enhance the sensitivity of detecting the engagement of neural circuits in cognitive processing. The approach thus aims to capture effects related to the stimulus- and state-dependent input correlations and differences in the propagation of vascular dilation between neural columns (O’Herron et al., 2016; Butler et al., 2017) that would manifest as differences in the MEG-fMRI correlation patterns across experimental tasks. It should, however, be noted that our approach does not directly tell whether the circuit is more or less engaged during a task; the correlation-based measure can only reveal that the relationship between the applied proxies has changed. For example, our two proxies (MEG and fMRI) can be negatively or positively correlated, without indicating whether the amount of activity in the circuit has increased or decreased compared to the other conditions. In areas where one imaging method reveals only noise and the other detectable cortical activation, the task-wise correlations should not be significant. Our clustering approach corresponds to a conditioning which enforces the method to consider only those correlations that are related to the performed cognitive tasks. In the optimal situation, both proxies would have similar temporal granularity but, due to the highly integrative nature of the fMRI signal, precise temporal information was present only in the MEG signals. Nonetheless, we can track and utilize the temporal information in the MEG signals to dissociate even subtle effects in the integrative fMRI signals and, thereby, discover also small differences between cognitive tasks.

Spatially, our clustering-based analysis was designed to identify robust, large-scale effects in the correlation patterns that were specific to the given three naming tasks. Thus, the clustering results may not necessarily obey conventional knowledge about the locations of task-relevant functional brain regions. The reason is that the clustering is constructed using a very limited set of tasks. If these tasks do not distinguish between certain brain regions, then those regions will fall into the same cluster. Moreover, if the spatial extent of a cluster is too large, even a relatively strong signal may be masked by other contradicting signals or noise originating from the same cluster. If a cluster is too small, weak but significant signals may disappear as the region of activation has been split into parts. Some of the clusters are necessarily non-informative because none of the brain regions—including inactive regions—are left out in a clustering process.

In our study, we let the method cluster both hemispheres together. Thus, it is also possible that in some cases weaker, interesting signals might have been masked by stronger signals from the other hemisphere. Such a scenario could be avoided by conducting separate clustering for each hemisphere; however, this might hide some of the inter-hemispheric effects that were detected with the present approach.

Conclusion

We introduced a correlation-based data-fusion analysis pipeline that utilizes two proxies of brain activity to enhance sensitivity for detecting the engagement of neural circuits in cognitive processing. Our results demonstrate that the approach discovers spatially, spectrally, and temporally unique task-specific information on cortical processing during picture naming. Multimodal data fusion based on correlations between electromagnetic and hemodynamic activity can thus reveal task-dependent neural engagement that may not be detected using the proxies of brain activity offered by one imaging method alone.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: The MEG and fMRI data cannot be made openly available, according to the ethical permission and national privacy regulations at the time of the study, but are available from the corresponding author on reasonable request and with permission of the Ethics Committee of the Hospital district of Helsinki and Uusimaa. Requests to access these datasets should be directed to TM, tommi.mononen@helsinki.fi.

Ethics statement

The studies involving human participants were reviewed and approved by the Ethics Committee of the Hospital district of Helsinki and Uusimaa. The patients/participants provided their written informed consent to participate in this study.

Author contributions

TM, JK, ML, EL, SK, and RS: conceptualization and writing—review and editing. TM, JK, ML, EL, and SK: methodology. TM, JK, and ML: validation and formal analysis. ML: investigation. TM, JK, ML, and RS: writing—original draft. SK and RS: supervision and funding acquisition. All authors contributions is based on the CRediT taxonomy.

Funding

This work was financially supported by the Academy of Finland (Finnish Center of Excellence in Computational Inference Research COIN and grants #292334, #294238 to SK; #255349, #315553 to RS; #257576 to JK; #286405 funding for TM), the Sigrid Jusélius Foundation (grant to RS), the Finnish Cultural Foundation (grant to ML), the Swedish Cultural Foundation in Finland (grant to ML), the Maud Kuistila Memorial Foundation (grant to ML), and Aalto Brain Center.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Brandman, T., and Peelen, M. V. (2017). Interaction between scene and object processing revealed by human fMRI and MEG decoding. J. Neurosci. 37, 7700–7710. doi: 10.1523/JNEUROSCI.0582-17.2017

PubMed Abstract | CrossRef Full Text | Google Scholar

Brookes, M. J., Vrba, J., Robinson, S. E., Stevenson, C. M., Peters, A. M., Barnes, G. R., et al. (2008). Optimising experimental design for MEG beamformer imaging. Neuroimage 39, 1788–1802. doi: 10.1016/j.neuroimage.2007.09.050

PubMed Abstract | CrossRef Full Text | Google Scholar

Butler, R., Bernier, P. M., Lefebvre, J., Gilbert, G., and Whittingstall, K. (2017). Decorrelated input dissociates narrow band gamma power and BOLD in human visual cortex. J. Neurosci. 37, 5408–5418. doi: 10.1523/JNEUROSCI.3938-16.2017

PubMed Abstract | CrossRef Full Text | Google Scholar

Calhoun, V. D., and Sui, J. (2016). Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 1, 230–244. doi: 10.1016/j.bpsc.2015.12.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Cichy, R. M., Pantazis, D., and Oliva, A. (2014). Resolving human object recognition in space and time. Nat. Neurosci. 17, 455–462. doi: 10.1038/nn.3635

PubMed Abstract | CrossRef Full Text | Google Scholar

Cichy, R. M., Pantazis, D., and Oliva, A. (2016). Similarity-based fusion of MEG and fMRI reveals spatio-temporal dynamics in human cortex during visual object recognition. Cereb. Cortex 26, 3563–3579. doi: 10.1093/cercor/bhw135

PubMed Abstract | CrossRef Full Text | Google Scholar

Conner, C. R., Ellmore, T. M., Pieters, T. A., Disano, M. A., and Tandon, N. (2011). Variability of the relationship between electrophysiology and BOLD-fMRI across cortical regions in humans. J. Neurosci. 31, 12855–12865. doi: 10.1523/JNEUROSCI.1457-11.2011

PubMed Abstract | CrossRef Full Text | Google Scholar

Cottereau, B. R., Ales, J. M., and Norcia, A. M. (2015). How to use fMRI functional localizers to improve EEG/MEG source estimation. J. Neurosci. Methods 250, 64–73. doi: 10.1016/j.jneumeth.2014.07.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Dale, A. M., Liu, A. K., Fischl, B. R., Buckner, R. L., Belliveau, J. W., Lewine, J. D., et al. (2000). Dynamic statistical parametric mapping: Combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron 26, 55–67. doi: 10.1016/S0896-6273(00)81138-1

CrossRef Full Text | Google Scholar

Destrieux, C., Fischl, B., Dale, A., and Halgren, E. (2010). Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage 53, 1–15. doi: 10.1016/j.neuroimage.2010.06.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Stat. 7, 1–26. doi: 10.1214/aos/1176344552

CrossRef Full Text | Google Scholar

Ekstrom, A. (2010). How and when the fMRI BOLD signal relates to underlying neural activity: The danger in dissociation. Brain Res. Rev. 62, 233–244. doi: 10.1016/j.brainresrev.2009.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Engemann, D. A., Kozynets, O., Sabbagh, D., Lemaitre, G., Varoquaux, G., Liem, F., et al. (2020). Combining magnetoencephalography with magnetic resonance imaging enhances learning of surrogate-biomarkers. Elife 9, 1–33. doi: 10.7554/eLife.54055

PubMed Abstract | CrossRef Full Text | Google Scholar

Fischl, B. (2012). FreeSurfer. Neuroimage 62, 774–781. doi: 10.1016/j.neuroimage.2012.01.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Fischl, B., Sereno, M. I., and Dale, A. M. (1999). Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system. Neuroimage 9, 195–207. doi: 10.1006/nimg.1998.0396

PubMed Abstract | CrossRef Full Text | Google Scholar

Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., et al. (2013). MEG and EEG data analysis with MNE-Python. Front. Neurosci. 7:267. doi: 10.3389/fnins.2013.00267

PubMed Abstract | CrossRef Full Text | Google Scholar

Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., et al. (2014). MNE software for processing MEG and EEG data. Neuroimage 86, 446–460. doi: 10.1016/j.neuroimage.2013.10.027

PubMed Abstract | CrossRef Full Text | Google Scholar

Hardoon, D. R., Ettinger, U., Mourao-Miranda, J., Antonova, E., Collier, D., Kumari, V., et al. (2009). Correlation-based multivariate analysis of genetic influence on brain volume. Neurosci. Lett. 450, 281–286. doi: 10.1016/j.neulet.2008.11.035

PubMed Abstract | CrossRef Full Text | Google Scholar

Henson, R. N., Flandin, G., Friston, K. J., and Mattout, J. (2010). A parametric empirical Bayesian framework for fMRI-constrained MEG/EEG source reconstruction. Hum. Brain Mapp. 31, 1512–1531. doi: 10.1002/hbm.20956

PubMed Abstract | CrossRef Full Text | Google Scholar

Hipp, J. F., and Siegel, M. (2015). BOLD fMRI correlation reflects frequency-specific neuronal correlation. Curr. Biol. 25, 1368–1374. doi: 10.1016/j.cub.2015.03.049

PubMed Abstract | CrossRef Full Text | Google Scholar

Iannaccone, R., Hauser, T. U., Staempfli, P., Walitza, S., Brandeis, D., and Brem, S. (2015). Conflict monitoring and error processing: New insights from simultaneous EEG–fMRI. Neuroimage 105, 395–407. doi: 10.1016/j.neuroimage.2014.10.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Jiang, F., Jin, H., Gao, Y., Xie, X., Cummings, J., Raj, A., et al. (2022). Time-varying dynamic network model for dynamic resting state functional connectivity in fMRI and MEG imaging. Neuroimage 254:119131. doi: 10.1016/j.neuroimage.2022.119131

PubMed Abstract | CrossRef Full Text | Google Scholar

Kujala, J., Jung, J., Bouvard, S., Lecaignard, F., Lothe, A., Bouet, R., et al. (2015). Gamma oscillations in V1 are correlated with GABAA receptor density: A multi-modal MEG and Flumazenil-PET study. Sci. Rep. 5:16347. doi: 10.1038/srep16347

PubMed Abstract | CrossRef Full Text | Google Scholar

Kujala, J., Sudre, G., Vartiainen, J., Liljestrom, M., Mitchell, T., and Salmelin, R. (2014). Multivariate analysis of correlation between electrophysiological and hemodynamic responses during cognitive processing. Neuroimage 92, 207–216. doi: 10.1016/j.neuroimage.2014.01.057

PubMed Abstract | CrossRef Full Text | Google Scholar

Laaksonen, H., Kujala, J., and Salmelin, R. (2008). A method for spatiotemporal mapping of event-related modulation of cortical rhythmic activity. Neuroimage 42, 207–217. doi: 10.1016/j.neuroimage.2008.04.175

PubMed Abstract | CrossRef Full Text | Google Scholar

Lankinen, K., Saari, J., Hlushchuk, Y., Tikka, P., Parkkonen, L., Hari, R., et al. (2018). Consistency and similarity of MEG- and fMRI-signal time courses during movie viewing. Neuroimage 173, 361–369. doi: 10.1016/j.neuroimage.2018.02.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Lauritzen, M., Mathiesen, C., Schaefer, K., and Thomsen, K. J. (2012). Neuronal inhibition and excitation, and the dichotomic control of brain hemodynamic and oxygen responses. Neuroimage 62, 1040–1050. doi: 10.1016/j.neuroimage.2012.01.040

PubMed Abstract | CrossRef Full Text | Google Scholar

Le Floch, E., Guillemot, V., Frouin, V., Pinel, P., Lalanne, C., Trinchera, L., et al. (2012). Significant correlation between a set of genetic polymorphisms and a functional brain network revealed by feature selection and sparse Partial Least Squares. Neuroimage 63, 11–24. doi: 10.1016/j.neuroimage.2012.06.061

PubMed Abstract | CrossRef Full Text | Google Scholar

Leonardelli, E., and Fairhall, S. L. (2022). Similarity-based fMRI-MEG fusion reveals hierarchical organisation within the brain’s semantic system. Neuroimage 259:119405. doi: 10.1016/j.neuroimage.2022.119405

PubMed Abstract | CrossRef Full Text | Google Scholar

Liljeström, M., Tarkiainen, A., Parviainen, T., Kujala, J., Numminen, J., Hiltunen, J., et al. (2008). Perceiving and naming actions and objects. NeuroImage 41, 1132–1141. doi: 10.1016/j.neuroimage.2008.03.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Liljeström, M., Hultén, A., Parkkonen, L., and Salmelin, R. (2009). Comparing MEG and fMRI views to naming actions and objects. Hum. Brain Mapp. 30, 1845–1856. doi: 10.1002/hbm.20785

PubMed Abstract | CrossRef Full Text | Google Scholar

Liljeström, M., Stevenson, C., Kujala, J., and Salmelin, R. (2015b). Task- and stimulus-related cortical networks in language production: Exploring similarity of MEG- and fMRI-derived functional connectivity. Neuroimage 120, 75–87. doi: 10.1016/j.neuroimage.2015.07.017

PubMed Abstract | CrossRef Full Text | Google Scholar

Liljeström, M., Kujala, J., Stevenson, C., and Salmelin, R. (2015a). Dynamic reconfiguration of the language network preceding onset of speech in picture naming. Hum. Brain Mapp. 36, 1202–1216. doi: 10.1002/hbm.22697

PubMed Abstract | CrossRef Full Text | Google Scholar

Logothetis, N. K. (2008). What we can do and what we cannot do with fMRI. Nature 453, 869–878. doi: 10.1038/nature06976

PubMed Abstract | CrossRef Full Text | Google Scholar

Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., and Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157. doi: 10.1038/35084005

PubMed Abstract | CrossRef Full Text | Google Scholar

Matchin, W., Brodbeck, C., Hammerly, C., and Lau, E. (2019). The temporal dynamics of structure and content in sentence comprehension: Evidence from fMRI-constrained MEG. Hum. Brain Mapp. 40, 663–678. doi: 10.1002/hbm.24403

PubMed Abstract | CrossRef Full Text | Google Scholar

Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., and Malach, R. (2005). Coupling between neuronal firing, field potentials, and FMRI in human auditory cortex. Science 309, 951–954. doi: 10.1126/science.1110913

PubMed Abstract | CrossRef Full Text | Google Scholar

Muthukumaraswamy, S. D., Edden, R. A., Jones, D. K., Swettenham, J. B., and Singh, K. D. (2009). Resting GABA concentration predicts peak gamma frequency and fMRI amplitude in response to visual stimulation in humans. Proc. Natl. Acad. Sci. U.S.A. 106, 8356–8361. doi: 10.1073/pnas.0900728106

PubMed Abstract | CrossRef Full Text | Google Scholar

O’Herron, P., Chhatbar, P. Y., Levy, M., Shen, Z., Schramm, A. E., Lu, Z., et al. (2016). Neural correlates of single-vessel haemodynamic responses in vivo. Nature 534, 378–382. doi: 10.1038/nature17965

PubMed Abstract | CrossRef Full Text | Google Scholar

Pearlson, G. D., Liu, J., and Calhoun, V. D. (2015). An introductory review of parallel independent component analysis (p-ICA) and a guide to applying p-ICA to genetic data and imaging phenotypes to identify disease-associated biological pathways and systems in common complex disorders. Front. Genet. 6:276. doi: 10.3389/fgene.2015.00276

PubMed Abstract | CrossRef Full Text | Google Scholar

Pisauro, M. A., Fouragnan, E., Retzler, C., and Philiastides, M. G. (2017). Neural correlates of evidence accumulation during value-based decisions revealed via simultaneous EEG-fMRI. Nat. Commun. 8:15808. doi: 10.1038/ncomms15808

PubMed Abstract | CrossRef Full Text | Google Scholar

Purcell, S. M., Wray, N. R., Stone, J. L., Visscher, P. M., O’donovan, M. C., Sullivan, P. F., et al. (2009). Common polygenic variation contributes to risk of schizophrenia and bipolar disorder. Nature 460, 748–752. doi: 10.1038/nature08185

PubMed Abstract | CrossRef Full Text | Google Scholar

Renvall, H., Formisano, E., Parviainen, T., Bonte, M., Vihla, M., and Salmelin, R. (2012a). Parametric merging of MEG and fMRI reveals spatiotemporal differences in cortical processing of spoken words and environmental sounds in background noise. Cereb. Cortex 22, 132–143. doi: 10.1093/cercor/bhr095

PubMed Abstract | CrossRef Full Text | Google Scholar

Renvall, H., Salmela, E., Vihla, M., Illman, M., Leinonen, E., Kere, J., et al. (2012b). Genome-wide linkage analysis of human auditory cortical activation suggests distinct loci on chromosomes 2, 3, and 8. J. Neurosci. 32, 14511–14518. doi: 10.1523/JNEUROSCI.1483-12.2012

PubMed Abstract | CrossRef Full Text | Google Scholar

Salmela, E., Renvall, H., Kujala, J., Hakosalo, O., Illman, M., Vihla, M., et al. (2016). Evidence for genetic regulation of the human parieto-occipital 10-Hz rhythmic activity. Eur. J. Neurosci. 44, 1963–1971. doi: 10.1111/ejn.13300

PubMed Abstract | CrossRef Full Text | Google Scholar

Sanders, J. A., Lewine, J. D., and Orrison, W. W. Jr. (1996). Comparison of primary motor cortex localization using functional magnetic resonance imaging and magnetoencephalography. Hum. Brain Mapp. 4, 47–57. doi: 10.1002/(SICI)1097-0193(1996)4:1<47::AID-HBM3>3.0.CO;2-P

CrossRef Full Text | Google Scholar

Scheeringa, R., Fries, P., Petersson, K. M., Oostenveld, R., Grothe, I., Norris, D. G., et al. (2011). Neuronal dynamics underlying high- and low-frequency EEG oscillations contribute independently to the human BOLD signal. Neuron 69, 572–583. doi: 10.1016/j.neuron.2010.11.044

PubMed Abstract | CrossRef Full Text | Google Scholar

Scheeringa, R., Koopmans, P. J., van Mourik, T., Jensen, O., and Norris, D. G. (2016). The relationship between oscillatory EEG activity and the laminar-specific BOLD signal. Proc. Natl. Acad. Sci. U.S.A. 113, 6761–6766. doi: 10.1073/pnas.1522577113

PubMed Abstract | CrossRef Full Text | Google Scholar

Schulz, M., Chau, W., Graham, S. J., Mcintosh, A. R., Ross, B., Ishii, R., et al. (2004). An integrative MEG-fMRI study of the primary somatosensory cortex using cross-modal correspondence analysis. Neuroimage 22, 120–133. doi: 10.1016/j.neuroimage.2003.10.049

PubMed Abstract | CrossRef Full Text | Google Scholar

Schwarzkopf, D. S., Robertson, D. J., Song, C., Barnes, G. R., and Rees, G. (2012). The frequency of visually induced gamma-band oscillations depends on the size of early human visual cortex. J. Neurosci. 32, 1507–1512. doi: 10.1523/JNEUROSCI.4771-11.2012

PubMed Abstract | CrossRef Full Text | Google Scholar

Sepulcre, J., Masdeu, J. C., Pastor, M. A., Goni, J., Barbosa, C., Bejarano, B., et al. (2009). Brain pathways of verbal working memory: A lesion-function correlation study. Neuroimage 47, 773–778. doi: 10.1016/j.neuroimage.2009.04.054

PubMed Abstract | CrossRef Full Text | Google Scholar

Singh, K. D. (2012). Which “neural activity” do you mean? fMRI, MEG, oscillations and neurotransmitters. Neuroimage 62, 1121–1130. doi: 10.1016/j.neuroimage.2012.01.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Stevenson, C. M., Wang, F., Brookes, M. J., Zumer, J. M., Francis, S. T., and Morris, P. G. (2012). Paired pulse depression in the somatosensory cortex: Associations between MEG and BOLD fMRI. Neuroimage 59, 2722–2732. doi: 10.1016/j.neuroimage.2011.10.037

PubMed Abstract | CrossRef Full Text | Google Scholar

Stippich, C., Freitag, P., Kassubek, J., Soros, P., Kamada, K., Kober, H., et al. (1998). Motor, somatosensory and auditory cortex localization by fMRI and MEG. Neuroreport 9, 1953–1957. doi: 10.1097/00001756-199806220-00007

PubMed Abstract | CrossRef Full Text | Google Scholar

Taulu, S., and Simola, J. (2006). Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Phys. Med. Biol. 51, 1759–1768. doi: 10.1088/0031-9155/51/7/008

CrossRef Full Text | Google Scholar

Thirion, B., Varoquaux, G., Dohmatob, E., and Poline, J. B. (2014). Which fMRI clustering gives good brain parcellations? Front. Neurosci. 8:167. doi: 10.3389/fnins.2014.00167

PubMed Abstract | CrossRef Full Text | Google Scholar

Vartiainen, J., Liljeström, M., Koskinen, M., Renvall, H., and Salmelin, R. (2011). Functional magnetic resonance imaging blood oxygenation level-dependent signal and magnetoencephalography evoked responses yield different neural functionality in reading. J. Neurosci. 31, 1048–1058. doi: 10.1523/JNEUROSCI.3113-10.2011

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, Y., and Holland, S. K. (2022). Bayesian MEG time courses with fMRI priors. Brain Imaging Behav. 16, 781–791. doi: 10.1007/s11682-021-00550-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Warbrick, T. (2022). Simultaneous EEG-fMRI: What have we learned and what does the future hold? Sensors 22:2262. doi: 10.3390/s22062262

PubMed Abstract | CrossRef Full Text | Google Scholar

Ward, J. H. Jr. (1963). Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58, 236–244. doi: 10.1080/01621459.1963.10500845

CrossRef Full Text | Google Scholar

Whitman, J. C., Ward, L. M., and Woodward, T. S. (2013). Patterns of cortical oscillations organize neural activity into whole-brain functional networks evident in the fMRI BOLD signal. Front. Hum. Neurosci. 7:80. doi: 10.3389/fnhum.2013.00080

PubMed Abstract | CrossRef Full Text | Google Scholar

Whittingstall, K., Stroink, G., and Schmidt, M. (2007). Evaluating the spatial relationship of event-related potential and functional MRI sources in the primary visual cortex. Hum. Brain Mapp. 28, 134–142. doi: 10.1002/hbm.20265

PubMed Abstract | CrossRef Full Text | Google Scholar