基于嵌入层手术与任务级波束搜索的Whisper持续学习
Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper
AI总结:
针对多语言ASR持续学习中的灾难性遗忘问题,提出嵌入层手术加任务级波束搜索的方法,在Whisper模型上实现预训练语言词错误率下降且不影响新增语言性能。
摘要:
当前多语言自动语音识别(ASR)模型仅支持世界上的一小部分语言。持续学习(CL)旨在解决这一问题,即在预训练模型中新增语言的同时,避免模型在已有语言上的性能下降,也就是所谓的灾难性遗忘(CF)。然而,现有持续学习方法忽略了解码器端词元嵌入查找表的适配,尽管它是导致灾难性遗忘的重要因素。我们提出嵌入层手术方法:为每种新语言创建独立的词元嵌入副本,转写对应新语言时选择其中一个副本替换旧语言嵌入。但该方法会因语言识别(LID)错误导致ASR嵌入选择错误,我们的任务级波束搜索可对这类错误进行自我修正。我们用Common Voice数据集中10种未见语言各10小时的数据适配Whisper模型,结果表明:与经验回放方法相比,我们的方法将预训练语言的平均词错误率(AWER)从14.2%降至11.9%,且不会损害未见语言的AWER。
英文摘要:
Current Multilingual ASR models only support a fraction of the world's languages. Continual Learning (CL) aims to tackle this problem by adding new languages to pre-trained models while avoiding the loss of performance on existing languages, also known as Catastrophic Forgetting (CF). However, existing CL methods overlook the adaptation of the token embedding lookup table at the decoder, despite its significant contribution to CF. We propose Embedding Layer Surgery where separate copies of the token embeddings are created for each new languages, and one of the copies is selected to replace the old languages embeddings when transcribing the corresponding new language. Unfortunately, this approach means LID errors also cause incorrect ASR embedding selection. Our Task-wise Beam Search allows self-correction for such mistakes. By adapting Whisper to 10 hours of data for each of 10 unseen languages from Common Voice, results show that our method reduces the Average WER (AWER) of pre-trained languages from 14.2% to 11.9% compared with Experience Replay, without compromising the AWER of the unseen languages.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。