arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.00508eess.AScs.CLcs.SD
收藏

面向知识提取的多码本矢量量化指标预测

Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation

Xiaomi Corp.(小米公司);Meaning.Team Inc(Meaning.Team公司)

Liyong Guo, Xiaoyu Yang, Quandong Wang, Yuxiang Kong, Zengwei Yao, Fan Cui, Fangjun Kuang, Wei Kang, Long Lin, Mingshuang Luo, Piotr Zelasko, Daniel Povey

摘要:

知识蒸馏(KD)是在自动语音识别(ASR)中提高模型性能的常用方法,其中学生模型被训练成模仿教师模型的输出行为。然而,传统的知识库方法存在教师标签存储问题,特别是当训练语料库很大时。虽然动态教师标签生成解决了这个问题,但训练速度明显较慢,因为教师模型必须在每个批次进行评估。在本文中,我们重新制定的教师标签的生成作为一个编解码器的问题。我们提出了一种新的多码本矢量量化(MVQ)的方法,压缩教师嵌入码本索引(CI)。在此基础上,提出了一个KD训练框架(MVQ-KD),其中学生模型预测从自监督预训练教师模型的嵌入中生成的CI。在LibriSpeech clean-100 hour上的实验表明,MVQ-KD框架实现了与传统KD方法(l1,l2)相当的性能,同时需要的存储量减少了256倍。当使用完整的LibriSpeech数据集时,MVQ-KD框架导致非流式转换器在测试-干净和测试-其他上的相对字错误率降低13.8%和8.2%,以及流式转换器的4.0%和4.9%。这项工作的实现已经作为开源项目icefall的一部分发布。

英文摘要:

Knowledge distillation(KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model. However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large. Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch. In this paper, we reformulate the generation of teacher label as a codec problem. We propose a novel Multi-codebook Vector Quantization (MVQ) approach that compresses teacher embeddings to codebook indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where a student model predicts the CI generated from the embeddings of a self-supervised pre-trained teacher model. Experiments on the LibriSpeech clean-100 hour show that MVQ-KD framework achieves comparable performance as traditional KD methods (l1, l2), while requiring 256 times less storage. When the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and 8.2% relative word error rate reductions (WERRs) for non -streaming transducer on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The implementation of this work is already released as a part of the open-source project icefall.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑