arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01774eess.ASeess.IV
收藏

多模态并发说话人检测的视听方法

Audio-Visual Approach For Multimodal Concurrent Speaker Detection

Amit Eliav, Sharon Gannot

摘要:

并发说话人检测(CSD)是在音频信号中识别活动说话人及其重叠部分的任务,对于各种音频应用是必不可少的,包括会议转录、说话人日记和语音分离。这项研究提出了一种多模态深度学习方法,整合了音频和视觉信息。该模型利用早期融合策略,通过跨模态注意力机制将音频和视觉特征与可学习的[CLS]令牌相结合,以捕获关键的视听关系。该模型在两个真实世界的数据集上进行了广泛的评估,即已建立的AMI数据集和最近推出的EasyCom数据集。实验验证了多模态融合策略的有效性。消融研究进一步支持设计选择和模型的训练程序。由于这是第一个工作报告CSD的结果具有挑战性的EasyCom数据集,研究结果表明,在现实世界的情况下,拟议的多模态方法\ac{CSD}的潜力。

英文摘要:

Concurrent Speaker Detection (CSD), the task of identifying active speakers and their overlaps in an audio signal, is essential for various audio applications, including meeting transcription, speaker diarization, and speech separation. This study presents a multimodal deep learning approach that integrates audio and visual information. The proposed model utilizes an early fusion strategy, combining audio and visual features through cross-modal attention mechanisms with a learnable [CLS] token to capture key audio-visual relationships. The model is extensively evaluated on two real-world datasets, the established AMI dataset and the recently introduced EasyCom dataset. Experiments validate the effectiveness of the multimodal fusion strategy. An ablation study further supports the design choices and the model's training procedure. As this is the first work reporting CSD results on the challenging EasyCom dataset, the findings demonstrate the potential of the proposed multimodal approach for \ac{CSD} in real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑