arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16078cs.SDeess.AS
收藏

结合预训练 AV-HuBERT 与掩码与恢复策略的目标语音提取

Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy

The Chinese University of Hong Kong(香港中文大学)

Wenxuan Wu, Xueyuan Chen, Xixin Wu, Haizhou Li, Helen Meng

AI总结:

本文将预训练 AV-HuBERT 集成到音视频目标语音提取系统,并提出自监督的掩码与恢复(MAR)策略,在 VoxCeleb2 数据集上主客观指标均优于基线,验证了方法的有效性。

摘要:

音视频目标语音提取(AV-TSE)是机器人技术和众多音视频应用中的一项关键使能技术。AV-TSE 的挑战之一是如何在该过程中有效利用音视频同步信息。AV-HuBERT 可作为唇语识别的有效预训练模型,但尚未被 AV-TSE 采用。本文旨在探索将预训练 AV-HuBERT 集成到 AV-TSE 系统中的方法,有充分理由预期其能提升性能。为利用模态间和模态内的相关性,我们还提出了一种用于自监督学习的新型掩码与恢复(MAR)策略。在 VoxCeleb2 数据集上的实验结果表明,所提模型在主观和客观指标上均优于基线,表明预训练 AV-HuBERT 模型为目标语音提取提供了更丰富的视觉线索。此外,通过对比研究,我们确认所提出的掩码与恢复策略具有显著有效性。

英文摘要:

Audio-visual target speech extraction (AV-TSE) is one of the enabling technologies in robotics and many audio-visual applications. One of the challenges of AV-TSE is how to effectively utilize audio-visual synchronization information in the process. AV-HuBERT can be a useful pre-trained model for lip-reading, which has not been adopted by AV-TSE. In this paper, we would like to explore the way to integrate a pre-trained AV-HuBERT into our AV-TSE system. We have good reasons to expect an improved performance. To benefit from the inter and intra-modality correlations, we also propose a novel Mask-And-Recover (MAR) strategy for self-supervised learning. The experimental results on the VoxCeleb2 dataset show that our proposed model outperforms the baselines both in terms of subjective and objective metrics, suggesting that the pre-trained AV-HuBERT model provides more informative visual cues for target speech extraction. Furthermore, through a comparative study, we confirm that the proposed Mask-And-Recover strategy is significantly effective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑