AUTHOR=Chan Vincent Y., Jin Craig T., van Schaik André 

TITLE=Neuromorphic Audio-Visual Sensor Fusion on a Sound-Localising Robot

JOURNAL=Frontiers in Neuroscience

VOLUME=6

YEAR=2012

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2012.00021

DOI=10.3389/fnins.2012.00021

ISSN=1662-453X

ABSTRACT=<p>This paper presents the first robotic system featuring audio–visual (AV) sensor fusion with neuromorphic sensors. We combine a pair of silicon cochleae and a silicon retina on a robotic platform to allow the robot to learn sound localization through self motion and visual feedback, using an adaptive ITD-based sound localization algorithm. After training, the robot can localize sound sources (white or pink noise) in a reverberant environment with an RMS error of 4–5° in azimuth. We also investigate the AV source binding problem and an experiment is conducted to test the effectiveness of matching an audio event with a corresponding visual event based on their onset time. Despite the simplicity of this method and a large number of false visual events in the background, a correct match can be made 75% of the time during the experiment.</p>