arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.05233eess.AS
收藏

大规模UniSpeech:大规模语音识别数据集预训练方法的实证研究

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, Furu Wei

摘要:

最近,人们对自监督学习(SSL)产生了极大的兴趣,其中模型在大规模未标记数据上进行预训练,然后在小的标记数据集上进行微调。普遍的看法是,SSL有助于资源有限的任务,其中只有有限数量的标记数据可用。SSL的好处随着标记训练数据量的增加而不断减少。据我们所知,在SSL的研究中使用了最多几千小时的标记数据。相比之下,该行业通常使用数万小时的标记数据来为资源丰富的语言构建高精度的语音识别(ASR)系统。在这项研究中,我们接受挑战,调查SSL是否以及如何提高最先进的生产规模Transformer-Transducer模型的ASR准确性,该模型是用6.5万小时的匿名标记EN-US数据构建的。

英文摘要:

Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset. The common wisdom is that SSL helps resource-limited tasks in which only a limited amount of labeled data is available. The benefit of SSL keeps diminishing when the labeled training data amount increases. To our best knowledge, at most a few thousand hours of labeled data was used in the study of SSL. In contrast, the industry usually uses tens of thousands of hours of labeled data to build high-accuracy speech recognition (ASR) systems for resource-rich languages. In this study, we take the challenge to investigate whether and how SSL can improve the ASR accuracy of a state-of-the-art production-scale Transformer-Transducer model, which was built with 65 thousand hours of anonymized labeled EN-US data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑