Seeing helps hearing: A multi-modal dataset and a mamba-based dual branch parallel network for auditory attention decoding by Cunhang Fan instant download

Seeing helps hearing: A multi-modal dataset and a mamba-based dual branch parallel network for auditory attention decoding by Cunhang Fan instant download

EEG-based auditory attention decoding (AAD) aims to identify the attended speaker from the listener’s EEGsignals. Existing datasets mainly focus on auditory stimuli, ignoring real-world multi-modal inputs. To addressthis, a new multi-modal AAD dataset (MM-AAD) is constructed, representing the first dataset to include audio–visual stimuli. Additionally, prior studies mostly extract single-domain features, neglecting complementarytemporal and frequency domain information, which can perform well in the within-trial setting but poorly inthe cross-trial setting. Therefore, a framework called Mamba-based dual branch parallel network (M-DBPNet)is proposed, effectively fusing temporal and frequency domain features. By adding Mamba, temporal featuresin time sequence signals are better extracted. Experimental results show that Mamba enhances decodingperformance in the within-trial setting with fewer parameters and demonstrates strong generalization inthe cross-trial setting. Visualization analysis indicates that visual stimuli strengthen evoked responses andactivation in temporal and occipital lobes, enhancing auditory perception and decoding performance.