CueNet:通过跨模态线索挖掘与交互实现鲁棒的音频视觉说话人提取
CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction
AI总结:
本文提出CueNet,通过跨模态线索挖掘与交互提升音频视觉说话人提取的鲁棒性,验证了模型在不同视觉退化下的优越性能。
摘要:
音频视觉说话人提取因其无需预注册语音并利用视觉模态作为音频的补充而受到越来越多关注。尽管现有方法已取得显著成效,但降质视觉输入的问题却较少受到重视,尽管在现实场景中普遍存在。以往的解决方案主要涉及使用降质视觉数据进行训练。然而,视觉退化可能以多种不可预测的方式发生,使在训练期间模拟所有可能情况变得不切实际。本文旨在增强音频视觉说话人提取在受损视觉输入下的鲁棒性,而无需依赖降质视频进行训练。受人类感知机制的启发,我们提出了一种音频视觉学习器,将说话人信息、声音同步和语义同步作为不同的线索进行解耦。此外,我们设计了一个专用交互模块,有效整合这些线索,为说话人提取提供可靠的指导信号。大量实验验证了所提模型在各种视觉退化下的强大鲁棒性以及其在现有方法上的明显优势。
英文摘要:
Audio-visual speaker extraction has attracted increasing attention, as it removes the need for pre-registered speech and leverages the visual modality as a complement to audio. Although existing methods have achieved impressive performance, the issue of degraded visual inputs has received relatively little attention, despite being common in real-world scenarios. Previous attempts to address this problem have mainly involved training with degraded visual data. However, visual degradation can occur in many unpredictable ways, making it impractical to simulate all possible cases during training. In this paper, we aim to enhance the robustness of audio-visual speaker extraction against impaired visual inputs without relying on degraded videos during training. Inspired by observations from human perceptual mechanisms, we propose an audio-visual learner that disentangles speaker information, acoustic synchronisation, and semantic synchronisation as distinct cues. Furthermore, we design a dedicated interaction module that effectively integrates these cues to provide a reliable guidance signal for speaker extraction. Extensive experiments demonstrate the strong robustness of the proposed model under various visual degradations and its clear superiority over existing methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。