利用音视频数据缩小自监督语音模型中的多语言差距
Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
AI总结：
本文提出在双语自监督语音模型中引入有限视觉grounding，以缩小多语言与单语模型间的性能差距，在零样本音素判别上将差距从31.5%降至8.04%。
摘要：
自监督学习（SSL）在语音表示学习方面取得了显著进展。wav2vec 2.0 和 HuBERT 等模型在语音识别等任务中已取得最先进的结果，尤其是在单语场景下。然而，多语言 SSL 模型在每种单独语言上的表现往往不如其单语对应模型，特别是在语言数量较少的多语言场景中，例如双语场景。在这项工作中，我们研究了一种新方法，通过向双语语音 SSL 模型中引入有限的视觉 grounding 来缩小这一性能差距。我们的结果表明，视觉 grounding 对单语和双语模型都有益，其中对后者的增益尤为显著，将零样本音素判别上的多语言性能差距从仅音频模型的 31.5% 缩小到引入 grounding 后的 8.04%。
英文摘要：
Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
