XLAVS-R：面向噪声鲁棒语音感知的跨语言视听语音表示学习
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
AI总结：
针对多语言视听数据稀缺问题，提出 XLAVS-R 表示模型，基于纯音频多语言预训练并简化 AV 预训练，在 MuAViC 上显著提升含噪语音识别与翻译性能。
摘要：
语音识别和翻译系统在含噪输入上表现不佳，而此类输入在现实环境中十分常见。利用视觉信号增强这些系统，有望提升其对噪声的鲁棒性。然而，视听（AV）数据的可用数量有限，且所覆盖语言少于纯音频资源。为弥补这一差距，我们提出 XLAVS-R，这是一个跨语言视听语音表示模型，用于支持 100 多种语言的噪声鲁棒语音识别和翻译。该模型建立在纯音频多语言预训练之上，并简化现有预训练方案，旨在最大限度发挥有限多语言 AV 预训练数据的效益。在 MuAViC 基准上的广泛评估表明，XLAVS-R 在下游视听语音识别和翻译任务中表现强劲：在给定含噪 AV 输入时，其较此前最优方法最高提升 18.5% WER 和 4.7 BLEU，并能通过纯音频微调实现强大的零样本视听能力。
英文摘要：
Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources. To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes. Extensive evaluation on the MuAViC benchmark shows the strength of XLAVS-R on downstream audio-visual speech recognition and translation tasks, where it outperforms the previous state of the art by up to 18.5% WER and 4.7 BLEU given noisy AV inputs, and enables strong zero-shot audio-visual ability with audio-only fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
