arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07696eess.AS
收藏

基于多视角的音频视觉目标说话人提取

Multi-View Based Audio Visual Target Speaker Extraction

Wuhan University(武汉大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Peijun Yang, Zhan Jin, Juan Liu, Ming Li

AI总结:

本文提出多视角张量融合框架,通过多视角学习提升音频视觉目标说话人提取的性能与鲁棒性。

摘要:

音频视觉目标说话人提取(AVTSE)旨在利用相应的视觉线索从混合音频信号中分离出目标说话人的声音。尽管大多数现有的AVTSE方法仅依赖于正面视角视频,这种限制限制了它们在现实世界中的鲁棒性,因为在非正面视角普遍存在的情况下。此类视觉视角往往包含互补的发音信息,可以增强语音提取。在本文中,我们提出了一种新的框架,即多视角张量融合(MVTF),将多视角学习转化为单视角性能提升。在训练阶段,我们利用同步的多视角嘴唇视频通过MVTF学习跨视角相关性,其中成对的外积显式地建模不同视角输入嘴唇嵌入之间的乘法交互。在推理阶段,系统支持单视角和多视角输入。实验结果表明,在单视角输入的情况下,我们的框架利用多视角知识实现显著的性能提升,而在多视角模式下,它进一步提高了整体性能并增强了鲁棒性。我们的演示、代码和数据可在https://anonymous.4open.science/w/MVTF-Gridnet-209C/上获得。

英文摘要:

Audio-Visual Target Speaker Extraction (AVTSE) aims to separate a target speaker's voice from a mixed audio signal using the corresponding visual cues. While most existing AVTSE methods rely exclusively on frontal-view videos, this limitation restricts their robustness in real-world scenarios where non-frontal views are prevalent. Such visual perspectives often contain complementary articulatory information that could enhance speech extraction. In this work, we propose Multi-View Tensor Fusion (MVTF), a novel framework that transforms multi-view learning into single-view performance gains. During the training stage, we leverage synchronized multi-perspective lip videos to learn cross-view correlations through MVTF, where pairwise outer products explicitly model multiplicative interactions between different views of input lip embeddings. At the inference stage, the system supports both single-view and multi-view inputs. Experimental results show that in the single-view inputs, our framework leverages multi-view knowledge to achieve significant performance gains, while in the multi-view mode, it further improves overall performance and enhances the robustness. Our demo, code and data are available at https://anonymous.4open.science/w/MVTF-Gridnet-209C/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑