Cross-Modal Congruency Modulates Evidence Accumulation, Not Decision Thresholds

Natálie Brožová ^1* Lukas Vollmer ¹ Björn Kampa ² Christoph Kayser ³ Janina Fels ¹

• ¹ Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany
• ² Systems Neurophysiology Department, Institute of Zoology, RWTH AAchen, Aachen, Germany
• ³ Department of Cognitive Neuroscience, Faculty of Biology, University of Bielefeld, Bielefeld, North Rhine-Westphalia, Germany

Audiovisual cross-modal correspondences (CMCs) refer to the brain's inherent ability to subconsciously connect auditory and visual information. These correspondences reveal essential aspects of multisensory perception and influence behavioral performance, enhancing reaction times and accuracy. However, the impact of different types of CMCs—arising from statistical co-occurrences or shaped by semantic associations—on information processing and decision-making remains underexplored.This study utilizes the Implicit Association Test, where unisensory stimuli are sequentially presented and linked via CMCs within an experimental block by the specific response instructions (either congruent or incongruent).Behavioral data are integrated with EEG measurements through neurally informed drift-diffusion modeling to examine how neural activity across both auditory and visual trials is modulated by CMCs.Our findings reveal distinct neural components that differentiate between congruent and incongruent stimuli regardless of modality, offering new insights into the role of congruency in shaping multisensory perceptual decision-making. Two key neural stages were identified: an Early component enhancing sensory encoding in congruent trials and a Late component affecting evidence accumulation, particularly in incongruent trials. These results suggest that cross-modal congruency primarily influences the processing and accumulation of sensory information rather than altering decision thresholds.