Manuel Varlet
Tijl Grootswagers- 1The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, NSW, Australia
- 2School of Psychology, Western Sydney University, Sydney, NSW, Australia
- 3School of Computer, Data and Mathematical Sciences, Western Sydney University, Sydney, NSW, Australia
Hyperscanning, which enables the recording of brain activity from multiple individuals simultaneously, has been increasingly used to investigate the neuropsychological processes underpinning social interaction. Previous hyperscanning research has primarily focused on interbrain synchrony, demonstrating an enhanced alignment of brain waves across individuals during social interaction. However, using EEG hyperscanning simulations, we here show that interbrain synchrony has low sensitivity to information alignment across people. Surprisingly, interbrain synchrony remains largely unchanged despite manipulating whether two individuals are seeing same or different things at the same time. Furthermore, we show that hyperscanning recordings do contain indices of interpersonal information alignment and that they can be captured using representational analyses. These findings highlight major limitations of current hyperscanning research and offer a promising alternative for investigating interactive minds.
1 Introduction
Understanding how social interactions dynamically shape human mind and behavior and vice versa is a key question in cognitive neuroscience and psychology (Sebanz et al., 2006; Hari et al., 2015). Hyperscanning, which enables the recording of the brain activity of multiple individuals at the same time, has been increasingly used to address this question. Hyperscanning research revealed enhanced interbrain synchrony between individuals interacting socially, which has been argued to reflect resonant minds facilitating cooperation and communication (Dumas et al., 2010; Dikker et al., 2017; Pérez et al., 2017; Goldstein et al., 2018; Reindl et al., 2018). However, here we highlight an important limitation of interbrain synchrony measures by showing that these measures indexing the alignment of neural activity patterns across individuals have low sensitivity to their Information content. We show that representational analyses, the framework of Representational Similarity Analysis (RSA) in particular, is a promising alternative to address this issue and better index information alignment across individuals in hyperscanning research.
2 Interbrain synchrony does not measure information alignment
Interbrain synchrony has been investigated in hyperscanning research using a range of neuroimaging techniques including Electroencephalography (EEG), Magnetoencephalography (MEG), functional Magnetic Resonance Imaging (fMRI), and functional Near-InfraRed Spectroscopy (fNIRS). Numerous measures have been used to index interbrain synchrony in hyperscanning studies generally based on amplitude correlation and phase locking (Czeszumski et al., 2020). For EEG and MEG, enabling access to neural processing at faster time scales, these analyses have also focused on the amplitude and phase of neural oscillations, with larger emphases on activity in the alpha (8–12 Hz) and beta (13–30 Hz) bands (Dumas et al., 2010; Goldstein et al., 2018). See Czeszumski et al. (2020) for a comprehensive review of hyperscanning and interbrain synchrony measures.
While all these methods are relevant to measure synchrony in general, the processes captured by interbrain synchrony in the context of hyperscanning research remain unclear. The causal nature of this phenomenon has been the main source of concerns so far, whether interbrain synchrony directly drives synchronised mind and behavior or interbrain synchrony is simply caused by synchronised mind and behavior (Burgess, 2013; Hari et al., 2013; Hamilton, 2021; Novembre and Iannetti, 2021; Holroyd, 2022). If synchronised mind and behavior causes interbrain synchrony, then this suggests that there is no direct brain-to-brain coupling between individuals, and interbrain synchrony is the result of similar sensory and cognitive processes driven by shared environment and task. As pointed out by Hamilton (2021), a lack of direct brain-to-brain coupling may not be a critical issue because interbrain synchrony measures are still of interest for hyperscanning research if they provide insight into the alignment of sensory information and cognitive processes across individuals while socially interacting.
However, here we present results showing that interbrain synchrony measures do not effectively index information alignment between individuals because these measures are largely content agnostic. We leveraged a publicly available EEG dataset from the visual object recognition literature with 48 participants who were presented with 400 images from 40 categories (Shatek et al., 2022). All recordings were made individually with a 128-channel BrainVision actichamp EEG system. We used this dataset to run hyperscanning simulations to assess the sensitivity of interbrain synchrony to information alignment by comparing interbrain synchrony when two individuals see the same things (Same objects) vs. different things (Different objects) at the same time (Figures 1, 2).
Figure 1. (A) Same vs. different mental representations across two subjects induced by the presentation of same vs. different objects at the same time. (B) Stimuli presented to subjects individually in Shatek et al. (2022), including 40 different categories of objects, each with 10 different images presented 10 times, while EEG used here to run hyperscanning simulations was recorded. The 10 images are shown for the categories Bee and Bench, and 2 out of the 10 images for each of the remaining 38 categories are shown in the ‘Other’ column. (C) EEG evoked responses for two representative subjects (at channel POz) for the 40 categories of objects averaged across all images and repetitions highlighting the low magnitude of variations related to image content.
Figure 2. (A) 10,000 hyperscanning simulations based on individual 128-channel EEG recordings from Shatek et al. (2022), each simulation including 24 pseudo pairs randomly drawn from the 48 subjects available. (B) EEG data were average referenced, bandpass filtered between 0.1 and 100 Hz, and downsampled to 250 Hz. Amplitude and phase were then estimated for each channel and all 40 objects (averaged across all stimuli and repetitions within categories) from 6 to 46 Hz every 10 ms using a 500 ms sliding Hanning window yielding a 2 Hz frequency resolution (Oostenveld et al., 2011). (C) Objects presented to the two subjects of each pair were either kept the same (Same objects) or made different by shuffling their order (Different objects). (D) Interbrain synchrony (IBS) was computed at each 10 ms time step and frequency bin as the amplitude (Pearson) correlation and phase locking (relative phase mean vector length) between the two subjects of the pairs for all combinations of channels within a 510 ms sliding window from 0.2 to 1 s before/after stimulus onset. The resulting 128 × 128 channel interbrain connectivity matrices for each object were then averaged to obtain a single amplitude correlation and phase locking value at each time step and frequency bin for Same objects and Different objects. Interbrain connectivity matrices are from a representative pseudo pair (at 10 Hz and 200 ms after stimulus onset) and time-frequency maps correspond to the grand average of all pseudo pairs and simulations. (E) For Interbrain Representational Similarity Analysis (IRSA), amplitude correlation and phase locking were first calculated between all pairs of objects within a 510 ms sliding window from 0.2 to 1 s before/after stimulus onset to obtain dissimilarity matrices. Amplitude dissimilarity was calculated as 1 – Pearson correlation and phase dissimilarity was calculated as 1 – phase locking (relative phase mean vector length). We then computed the (Spearman) correlation between dissimilarity matrices across the two subjects for all combinations of channels. The resulting 128 × 128 channel interbrain connectivity matrices were averaged to obtain a single correlation value at each time step and frequency bin for Same objects and Different objects. The figure shows example dissimilarity matrices and interbrain connectivity matrices from a representative pseudo pair (at 10 Hz and 200 ms after stimulus onset, channel POz for the dissimilarity matrices). Time-frequency maps correspond to the grand average of all pseudo pairs and simulations. (F) Time-frequency maps (averaged across all possible combinations of channels) represent the percentage of significant t-values from the 10,000 simulations for Same objects vs. Different objects (t-values >1.714, critical t-value for one-tailed t-tests with 24 observations, alpha = 0.05) for amplitude and phase data as a function of IBS and IRSA, with brighter colours indicating that large proportions of simulations yielded significant differences. The right panels represent the mean t-values within 180–220 ms (time interval showing highest percentages of significant t-values in line with peak decoding accuracy in object literature) from the 10,000 simulations for Same objects vs. Different objects after averaging correlation and phase locking data across all frequency bins. The blue horizontal line indicates the critical t-value. These plots show that the same vs. different objects differences were observed only in around 50% of simulations using IBS, but in more than 80% of simulations when using IRSA.
We ran 10,000 hyperscanning simulations with 24 pseudo pairs randomly drawn from the 48 subjects available (Figure 2). Interbrain synchrony was computed for a wide range of frequencies as amplitude correlation and phase locking across subjects for all possible combinations of channels (Czeszumski et al., 2020). Results were averaged across all possible combinations of channels to capture any potential changes in interbrain synchrony. With optimal control of the timing and content of the presented stimuli, the results from the simulations revealed that interbrain synchrony has low sensitivity to information alignment across individuals. Amplitude correlation and phase locking values remained largely unchanged whether people were seeing the same object or not (Figure 2D). t-values after stimulus presentation across all frequencies obtained from the 10,000 simulations show that difference between Same objects and Different objects would in most cases be statistically undetectable despite a sample size at least similar (i.e., 24 pairs) to those generally used in the literature (Figure 2F). Figure 2F depicting t-values on averaged amplitude and phase values across frequencies within 180 and 220 ms after image presentation – time window yielding highest percentages of significant t-values and peak decoding accuracy in object literature in general (Grootswagers et al., 2019; Shatek et al., 2022) – shows that barely 50% of the simulations reached statistical significance for interbrain synchrony (IBS). See legend of Figure 2 and publicly available code at https://osf.io/etx64/ for further methodological details.
3 Interbrain representational similarity analysis to effectively measure information alignment
The Representational Similarity Analysis (RSA) framework has received significant interest in the field of cognitive neuroscience (Kriegeskorte et al., 2008; Haxby et al., 2014), including more recently in social neuroscience and the intersubject correlation community (Nastase et al., 2019; Popal et al., 2019), but has not been used yet in hyperscanning research. This analysis is based on the computation of Representational (Dis)similarity Matrices (RDMs) to abstract from the patterns of neural activity themselves and characterise their informational content, allowing testing how two different systems quantitatively relate to each other by comparing their RDMs. RSA makes it possible to compare the responses from different systems, including responses recorded with different neuroimaging systems, neuroimaging and behavioral responses, as well as responses from different individuals, as shown here.
We computed Interbrain RSA (IRSA) using the same amplitude and phase data as IBS (Figure 2E). RDMs that encode information content and abstract from activity patterns were first computed separately for each subject and channel by calculating the amplitude correlation and phase locking for all pairs of stimuli. This approach based on (dis)similar temporal patterns in time-frequency data separately for each channel differs from other approaches often used with RSA (Kriegeskorte et al., 2008; e.g., Grootswagers et al., 2017; Shatek et al., 2022), but allows here to directly compare between IRSA and IBS. The results of the simulations show that information alignment can be captured in EEG hyperscanning with better sensitivity using IRSA. IRSA values decrease dramatically when shuffling the stimuli (Figure 2E), as reflected in differences between Same object vs. Different objects being statistically detected more consistently than IBS with statistical significance being reached in most simulations (Figure 2F). See legend of Figure 2 and publicly available code at https://osf.io/etx64/ for further methodological details.
4 IBS vs. IRSA: statistical comparison
Statistical analyses on the 10,000 simulations indicated that t-values testing differences between Same vs. Different objects in the 180–220 ms selected time window, as depicted in Figure 2F, were significantly higher for IRSA than IBS for both amplitude, t(19998) = 117.22, p < 0.0001, d = 1.66, BF10 > 1,000, and phase, t(19998) = 109.31, p < 0.0001, d = 1.55, BF10 > 1,000, showing that IRSA has higher sensitivity to content shuffling than IBS. These effects held beyond the selected time window as indicated by significantly higher t-values for IRSA than IBS for both amplitude, t(19998) = 139.21, p < 0.0001, d = 1.97, BF10 > 1,000, and phase, t(19998) = 95.87, p < 0.0001, d = 1.36, BF10 > 1,000, when conducting these analyses on a larger time interval from 0 to 500 ms after stimulus presentation. Large effect sizes observed here demonstrate a critical advantage of IRSA over IBS to index information alignment from amplitude and phase data in hyperscanning research.
5 Perspectives and challenges
Using hyperscanning simulations with well-controlled visual stimuli, our work shows that content-related information in hyperscanning research is not effectively captured by interbrain synchrony measures. This contrasts with previous research that found reliable decrease in interbrain synchrony when shuffling data across trials and/or pairs (Zamm et al., 2021, 2024; Reindl et al., 2022), which our results suggest might most likely be due to breaking the alignment of the timing of sensory and cognitive processes occurring in shared tasks and environments rather than the alignment of their information content. Shuffling information content while keeping timing constant in our simulations only marginally decreased interbrain synchrony whereas changes in timing resulting in large variations in neural signals would have a strong influence (Burgess, 2013; Holroyd, 2022). Examining and controlling for timing (dis)alignment of sensory, cognitive, and motor responses when shuffling across pairs and/or trials will be essential in future research to better understand the processes reflected by interbrain synchrony.
More generally, these results support the hypothesis that interbrain synchrony is caused by synchronised mind and behavior rather than the opposite, interbrain synchrony causing synchronised mind and behavior. Showing that interbrain synchrony does not uniquely capture sensory and cognitive processes supporting social interactions, our results suggest that interbrain synchrony cannot be a causal mechanism (underpinned by direct brain-to-brain coupling), and importantly, might be at best a poor proxy of synchronised sensory and cognitive processes supporting social interactions (Hamilton, 2021; Novembre and Iannetti, 2021). To be an effective top-down mechanism enabling the myriads of social behaviors observed every day, interbrain synchrony would not only require reflecting reliable timing information but also content information, which is not supported by our hyperscanning simulations.
The lack of unique and direct relationship between interbrain synchrony and synchronised sensory and cognitive processes during social interactions might explain part of the discrepancies in hyperscanning literature, including interbrain synchrony not being observed in some studies despite participants interacting and synchronising with each other (Liu et al., 2018; Czeszumski et al., 2020; Newman et al., 2024). While further research with a wider range of sensory and cognitive processes and neuroimaging techniques will be needed to expand our work beyond EEG visual evoked responses, the limitations of interbrain synchrony revealed here add to the growing concerns having been expressed in the field (Burgess, 2013; Hari et al., 2013; Hamilton, 2021; Novembre and Iannetti, 2021; Holroyd, 2022).
Our results highlight representational analyses as a powerful alternative to synchrony measures to better index information alignment between individuals. Enabling information content to be compared across individuals, even while obtained from different (neuroimaging) systems, these analyses will have critical advantages for future hyperscanning research. While implementing representational analyses is relatively straightforward when having time-locked trials with many different stimuli, moving into the representational space in more naturistic tasks as often used in hyperscanning research will be more challenging. Advancing representational analyses methods for non-time locked data will be needed to reach full capacity of hyperscanning and enable a step change in understanding the sensory and cognitive processes supporting real-time social interactions.
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://osf.io/etx64/.
Ethics statement
The studies involving humans were approved by University of Sydney Human Research Ethics Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
MV: Conceptualization, Methodology, Writing – original draft. TG: Conceptualization, Methodology, Writing – review & editing.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research was supported by the Australian Research Council (DP220103047, DE230100380).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Burgess, A. P. (2013). On the interpretation of synchronization in EEG hyperscanning studies: a cautionary note. Front. Hum. Neurosci. 7:881. doi: 10.3389/fnhum.2013.00881
Czeszumski, A., Eustergerling, S., Lang, A., Menrath, D., Gerstenberger, M., Schuberth, S., et al. (2020). Hyperscanning: a valid method to study neural inter-brain underpinnings of social interaction. Front. Hum. Neurosci. 14:39. doi: 10.3389/fnhum.2020.00039
Dikker, S., Wan, L., Davidesco, I., Kaggen, L., Oostrik, M., McClintock, J., et al. (2017). Brain-to-brain synchrony tracks real-world dynamic group interactions in the classroom. Curr. Biol. 27, 1375–1380. doi: 10.1016/j.cub.2017.04.002
Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., and Garnero, L. (2010). Inter-brain synchronization during social interaction. PLoS One 5:e12166. doi: 10.1371/journal.pone.0012166
Goldstein, P., Weissman-Fogel, I., Dumas, G., and Shamay-Tsoory, S. G. (2018). Brain-to-brain coupling during handholding is associated with pain reduction. Proc. Natl. Acad. Sci. 115, E2528–E2537. doi: 10.1073/pnas.1703643115
Grootswagers, T., Robinson, A. K., and Carlson, T. A. (2019). The representational dynamics of visual objects in rapid serial visual processing streams. NeuroImage 188, 668–679. doi: 10.1016/j.neuroimage.2018.12.046
Grootswagers, T., Wardle, S. G., and Carlson, T. A. (2017). Decoding dynamic brain patterns from evoked responses: a tutorial on multivariate pattern analysis applied to time series neuroimaging data. J. Cogn. Neurosci. 29, 677–697. doi: 10.1162/jocn_a_01068
Hamilton, A. F. C. (2021). Hyperscanning: beyond the hype. Neuron 109, 404–407. doi: 10.1016/j.neuron.2020.11.008
Hari, R., Henriksson, L., Malinen, S., and Parkkonen, L. (2015). Centrality of social interaction in human brain function. Neuron 88, 181–193. doi: 10.1016/j.neuron.2015.09.022
Hari, R., Himberg, T., Nummenmaa, L., Hämäläinen, M., and Parkkonen, L. (2013). Synchrony of brains and bodies during implicit interpersonal interaction. Trends Cogn. Sci. 17, 105–106. doi: 10.1016/j.tics.2013.01.003
Haxby, J. V., Connolly, A. C., and Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annu. Rev. Neurosci. 37, 435–456. doi: 10.1146/annurev-neuro-062012-170325
Holroyd, C. B. (2022). Interbrain synchrony: on wavy ground. Trends Neurosci. 45, 346–357. doi: 10.1016/j.tins.2022.02.002
Kriegeskorte, N., Mur, M., and Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:4. doi: 10.3389/neuro.06.004.2008
Liu, D., Liu, S., Liu, X., Zhang, C., Li, A., Jin, C., et al. (2018). Interactive brain activity: review and progress on EEG-based hyperscanning in social interactions. Front. Psychol. 9:1862. doi: 10.3389/fpsyg.2018.01862
Nastase, S. A., Gazzola, V., Hasson, U., and Keysers, C. (2019). Measuring shared responses across subjects using intersubject correlation. Soc. Cogn. Affect. Neurosci. 14, 667–685. doi: 10.1093/scan/nsz037
Newman, L. A., Cao, M., Täuber, S., and van Vugt, M. (2024). Working memory load impairs tacit coordination but not inter-brain EEG synchronization. Soc. Cogn. Affect. Neurosci. 19:nsae017. doi: 10.1093/scan/nsae017
Novembre, G., and Iannetti, G. D. (2021). Hyperscanning alone cannot prove causality. Multibrain stimulation can. Trends Cogn. Sci. 25, 96–99. doi: 10.1016/j.tics.2020.11.003
Oostenveld, R., Fries, P., Maris, E., and Schoffelen, J.-M. (2011). FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci. 2011, 1–9. doi: 10.1155/2011/156869
Pérez, A., Carreiras, M., and Duñabeitia, J. A. (2017). Brain-to-brain entrainment: EEG interbrain synchronization while speaking and listening. Sci. Rep. 7, 1–12. doi: 10.1038/s41598-017-04464-4
Popal, H., Wang, Y., and Olson, I. R. (2019). A guide to representational similarity analysis for social neuroscience. Soc. Cogn. Affect. Neurosci. 14, 1243–1253. doi: 10.1093/scan/nsz099
Reindl, V., Gerloff, C., Scharke, W., and Konrad, K. (2018). Brain-to-brain synchrony in parent-child dyads and the relationship with emotion regulation revealed by fNIRS-based hyperscanning. NeuroImage 178, 493–502. doi: 10.1016/j.neuroimage.2018.05.060
Reindl, V., Wass, S., Leong, V., Scharke, W., Wistuba, S., Wirth, C. L., et al. (2022). Multimodal hyperscanning reveals that synchrony of body and mind are distinct in mother-child dyads. NeuroImage 251:118982. doi: 10.1016/j.neuroimage.2022.118982
Sebanz, N., Bekkering, H., and Knoblich, G. (2006). Joint action: bodies and minds moving together. Trends Cogn. Sci. 10, 70–76. doi: 10.1016/j.tics.2005.12.009
Shatek, S. M., Robinson, A. K., Grootswagers, T., and Carlson, T. A. (2022). Capacity for movement is an organisational principle in object representations. NeuroImage 261:119517. doi: 10.1016/j.neuroimage.2022.119517
Zamm, A., Loehr, J. D., Vesper, C., Konvalinka, I., Kappel, S. L., Heggli, O. A., et al. (2024). A practical guide to EEG hyperscanning in joint action research: from motivation to implementation. Soc. Cogn. Affect. Neurosci. 19:nsae026. doi: 10.1093/scan/nsae026
Keywords: hyperscanning, EEG, interbrain synchrony, social interaction, representational alignment, information content
Citation: Varlet M and Grootswagers T (2024) Measuring information alignment in hyperscanning research with representational analyses: moving beyond interbrain synchrony. Front. Hum. Neurosci. 18:1385624. doi: 10.3389/fnhum.2024.1385624
Edited by:
Markus Franziskus Mueller, Autonomous University of the State of Morelos, MexicoReviewed by:
Jorge Hermosillo Valadez, Autonomous University of the State of Morelos, MexicoCopyright © 2024 Varlet and Grootswagers. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Manuel Varlet, m.varlet@westernsydney.edu.au