arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.00733eess.AScs.CLcs.SD
收藏

UniWav:迈向语音表示学习与生成的统一预训练

UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);NVIDIA(英伟达公司)

Alexander H. Liu, Sang-gil Lee, Chao-Han Huck Yang, Yuan Gong, Yu-Chiang Frank Wang, James R. Glass, Rafael Valle, Bryan Catanzaro

AI总结:

本文提出统一预训练框架 UniWav,通过联合学习表示编码器和生成式音频解码器,在语音识别、文本到语音和语音分词任务上取得与专用基础模型相当的性能,证明了单一通用基础模型替代多个专用模型的可行性。

摘要:

预训练和表示学习在现代语音处理中正发挥着越来越重要的作用。然而,不同的应用一直依赖于不同的基础模型,因为主流的预训练技术要么专为判别式任务设计,要么专为生成式任务设计。在本研究中,我们首次尝试为语音中的两类任务构建一个统一的预训练框架。我们表明,通过适当的预训练设计选择,可以联合学习一个表示编码器和一个生成式音频解码器,并将其应用于这两类任务。我们提出了 UniWav,这是一个旨在统一预训练表示学习和生成式任务的编码器-解码器框架。在语音识别、文本到语音和语音分词任务上,UniWav 取得了与不同现有基础模型相当的性能,而这些现有模型均是在特定任务上训练的。我们的研究结果表明,可以构建一个用于语音的单一通用基础模型来替代不同的基础模型,从而降低预训练的开销和成本。

英文摘要:

Pre-training and representation learning have been playing an increasingly important role in modern speech processing. Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques are either designed for discriminative tasks or generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks. We propose UniWav, an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokenization, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general-purpose foundation model for speech can be built to replace different foundation models, reducing the overhead and cost of pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑