arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.01530cs.MM
收藏

CueNet:通过跨模态线索挖掘与交互实现鲁棒的音频视觉说话人提取

CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction

University Hospital rechts der Isar, Technical University of Munich(慕尼黑工业大学右岸医院);School of Artificial Intelligence, the Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)人工智能学院);Department of Computer Science and Technology, University of Science and Technology Beijing(北京科技大学计算机科学与技术系);Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电气与计算机工程系);Guangdong Provincial Key Laboratory of Big Data Computing(广东省大数据计算重点实验室);Shenzhen Research Institute of Big data(深圳大数据研究院);University of Bremen(不来梅大学);Imperial College London(帝国理工学院)

Jiadong Wang, Ke Zhang, Xinyuan Qian, Ruijie Tao, Haizhou Li, Björn Schuller

AI总结:

本文提出CueNet,通过跨模态线索挖掘与交互提升音频视觉说话人提取的鲁棒性,验证了模型在不同视觉退化下的优越性能。

摘要:

音频视觉说话人提取因其无需预注册语音并利用视觉模态作为音频的补充而受到越来越多关注。尽管现有方法已取得显著成效,但降质视觉输入的问题却较少受到重视,尽管在现实场景中普遍存在。以往的解决方案主要涉及使用降质视觉数据进行训练。然而,视觉退化可能以多种不可预测的方式发生,使在训练期间模拟所有可能情况变得不切实际。本文旨在增强音频视觉说话人提取在受损视觉输入下的鲁棒性,而无需依赖降质视频进行训练。受人类感知机制的启发,我们提出了一种音频视觉学习器,将说话人信息、声音同步和语义同步作为不同的线索进行解耦。此外,我们设计了一个专用交互模块,有效整合这些线索,为说话人提取提供可靠的指导信号。大量实验验证了所提模型在各种视觉退化下的强大鲁棒性以及其在现有方法上的明显优势。

英文摘要:

Audio-visual speaker extraction has attracted increasing attention, as it removes the need for pre-registered speech and leverages the visual modality as a complement to audio. Although existing methods have achieved impressive performance, the issue of degraded visual inputs has received relatively little attention, despite being common in real-world scenarios. Previous attempts to address this problem have mainly involved training with degraded visual data. However, visual degradation can occur in many unpredictable ways, making it impractical to simulate all possible cases during training. In this paper, we aim to enhance the robustness of audio-visual speaker extraction against impaired visual inputs without relying on degraded videos during training. Inspired by observations from human perceptual mechanisms, we propose an audio-visual learner that disentangles speaker information, acoustic synchronisation, and semantic synchronisation as distinct cues. Furthermore, we design a dedicated interaction module that effectively integrates these cues to provide a reliable guidance signal for speaker extraction. Extensive experiments demonstrate the strong robustness of the proposed model under various visual degradations and its clear superiority over existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑