arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04512cs.CLcs.SDeess.AS
收藏

蒸馏还是不蒸馏?论鲁棒知识蒸馏的鲁棒性

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

MBZUAI(穆罕默德·本·扎耶德人工智能大学);The University of British Columbia(不列颠哥伦比亚大学)

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

AI总结:

本文针对阿拉伯语ASR的方言多样性和高计算成本问题,通过知识蒸馏训练高效学生模型,并构建五方言评估数据集,实验表明蒸馏模型以45.0% WER超越更大规模的SoTA和教师模型。

摘要:

阿拉伯语被公认为给自动语音识别(ASR)带来了独特挑战。一方面,其丰富的语言多样性和广泛的方言种类使得开发稳健且具有包容性的模型变得复杂。另一方面,当前的多语言ASR模型计算密集,且缺乏适当的全面评估。鉴于这些挑战,我们将知识从大型教师模型蒸馏到更高效的小型学生变体中。我们还引入了一个新的人工标注数据集,涵盖五种代表性不足的阿拉伯方言用于评估。我们进一步在标准可用基准和我们新的方言数据上评估了我们的模型以及现有的最先进(SoTA)多语言模型。我们最佳蒸馏模型的整体性能(45.0% WER)超过了规模为其两倍的SoTA模型(SeamlessM4T-large-v2,WER=47.0%)及其教师模型(Whisper-large-v2,WER=55.1%),并且其在我们新方言数据上的平均性能(56.9% WER)优于所有其他模型。为了更深入地了解这些模型在方言数据上表现不佳的原因,我们进行了错误分析,并报告了不同模型倾向于产生的主要错误类型。该项目的GitHub仓库可在https://github.com/UBC-NLP/distill-whisper-ar获取。

英文摘要:

Arabic is known to present unique challenges for Automatic Speech Recognition (ASR). On one hand, its rich linguistic diversity and wide range of dialects complicate the development of robust, inclusive models. On the other, current multilingual ASR models are compute-intensive and lack proper comprehensive evaluations. In light of these challenges, we distill knowledge from large teacher models into smaller student variants that are more efficient. We also introduce a novel human-annotated dataset covering five under-represented Arabic dialects for evaluation. We further evaluate both our models and existing SoTA multilingual models on both standard available benchmarks and our new dialectal data. Our best-distilled model's overall performance ($45.0$\% WER) surpasses that of a SoTA model twice its size (SeamlessM4T-large-v2, WER=$47.0$\%) and its teacher model (Whisper-large-v2, WER=$55.1$\%), and its average performance on our new dialectal data ($56.9$\% WER) outperforms all other models. To gain more insight into the poor performance of these models on dialectal data, we conduct an error analysis and report the main types of errors the different models tend to make. The GitHub repository for the project is available at \url{https://github.com/UBC-NLP/distill-whisper-ar}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑