arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00837cs.CLcs.AIcs.SDeess.AS
收藏

迈向面向数千种语言的鲁棒语音表示学习

Towards Robust Speech Representation Learning for Thousands of Languages

Carnegie Mellon University(卡内基梅隆大学);Shanghai Jiaotong University(上海交通大学);Toyota Technological Institute at Chicago(芝加哥丰田技术学院)

William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, Shinji Watanabe

AI总结:

提出覆盖4057种语言、训练数据超100万小时的XEUS语音编码器,通过去混响目标增强SSL鲁棒性,并在ML-SUPERB上取得新SOTA。

摘要:

自监督学习(SSL)通过减少对标注数据的需求,帮助将语音技术扩展到更多语言。然而,现有模型仍远不能支持世界上7000多种语言。我们提出XEUS,这是一个面向通用语音的跨语言编码器,在覆盖4057种语言的超过100万小时数据上训练,将SSL模型的语言覆盖范围扩大了4倍。我们将来自现有公开可访问语料库的100万小时语音,与一个新建的、来自4057种语言的7400多小时语料库相结合,该新语料库将公开发布。为处理多语言语音数据的多样条件,我们在典型的SSL掩码预测方法之外增加了一种新颖的去混响目标,以提升鲁棒性。我们在多个基准上评估XEUS,结果表明它在一系列任务上始终优于最先进(SOTA)SSL模型,或取得与之相当的结果。XEUS在ML-SUPERB基准上创造了新的SOTA:尽管参数或预训练数据更少,它仍分别比MMS 1B和w2v-BERT 2.0 v2高出0.8%和4.4%。检查点、代码和数据可在https://www.wavlab.org/activities/2024/xeus/获取。

英文摘要:

Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. However, models are still far from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual Encoder for Universal Speech, trained on over 1 million hours of data across 4057 languages, extending the language coverage of SSL models 4-fold. We combine 1 million hours of speech from existing publicly accessible corpora with a newly created corpus of 7400+ hours from 4057 languages, which will be publicly released. To handle the diverse conditions of multilingual speech data, we augment the typical SSL masked prediction approach with a novel dereverberation objective, increasing robustness. We evaluate XEUS on several benchmarks, and show that it consistently outperforms or achieves comparable results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or pre-training data. Checkpoints, code, and data are found in https://www.wavlab.org/activities/2024/xeus/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑