流式多语言ASR中面向尾部语言的高效适配器微调
Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR
AI总结：
本文提出语言依赖适配器（LDA）微调方法，在冻结的Conformer transducer基础模型上仅训练占每语言0.4%参数的适配器，结合教师伪标签，在39种尾部语言上平均降低12.2%词错误率，缓解多语言流式ASR的异步峰值性能问题。
摘要：
端到端ASR模型在流式多语言场景中通常是理想选择，因为它更易于部署，并且可以从预训练语音模型（例如强大的基础模型）中受益。与此同时，不同语言的异构性和数据丰度的不平衡可能导致性能下降，导致不同语言在训练期间达到峰值性能的时间不同步，尤其是尾部语言。有时，由于隐私保护增强，数据本身甚至可能变得不可用。现有工作倾向于显著增加模型规模或学习语言特定的解码器来分别适应每种语言。在本研究中，我们在级联Conformer transducer框架下探索了简单而有效的语言依赖适配器（LDA）微调，并通过教师伪标签增强，用于流式多语言ASR中的尾部语言。该适配器每种语言仅占完整模型的0.4%。它被插入到冻结的基础模型中，并且是在带噪学生训练（noisy student training）的微调过程中唯一可训练的模块。最终模型合并来自不同语言的不同检查点的适配器参数。模型性能在一个具有挑战性的多语言听写数据集上进行了验证，该数据集包含39种尾部语言，涵盖拉丁语、希腊语、阿拉伯语等。我们提出的方法平均带来12.2%的词错误率降低，在单个语言环境上最高可达37.5%。此外，我们表明我们的参数高效LDA可以达到与完整模型微调相当的质量，从而大大缓解了异步峰值性能问题。
英文摘要：
The end-to-end ASR model is often desired in the streaming multilingual scenario since it is easier to deploy and can benefit from pre-trained speech models such as powerful foundation models. Meanwhile, the heterogeneous nature and imbalanced data abundance of different languages may cause performance degradation, leading to asynchronous peak performance for different languages during training, especially on tail ones. Sometimes even the data itself may become unavailable as a result of the enhanced privacy protection. Existing work tend to significantly increase the model size or learn language-specific decoders to accommodate each language separately. In this study, we explore simple yet effective Language-Dependent Adapter (LDA) finetuning under a cascaded Conformer transducer framework enhanced by teacher pseudo-labeling for tail languages in the streaming multilingual ASR. The adapter only accounts for 0.4% of the full model per language. It is plugged into the frozen foundation model and is the only trainable module during the finetuning process with noisy student training. The final model merges the adapter parameters from different checkpoints for different languages. The model performance is validated on a challenging multilingual dictation dataset, which includes 39 tail languages across Latin, Greek, Arabic, etc. Our proposed method brings 12.2% word error rate reduction on average and up to 37.5% on a single locale. Furthermore, we show that our parameter-efficient LDA can match the quality of the full model finetuning, thus greatly alleviating the asynchronous peak performance issue.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
