arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00756eess.AScs.SD
收藏

更少遗忘以更好泛化:探索语音自监督表示的持续学习微调方法

Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations

Télécom Paris, Institut Polytechnique de Paris(巴黎综合理工学院电信学院);Samsung AI Research(三星人工智能研究院)

Salah Zaiem, Titouan Parcollet, Slim Essid

AI总结:

本文针对语音自监督模型微调中预训练鲁棒性受损的问题,提出在微调时引入持续学习方法以减少对自监督任务的遗忘,并在英语和丹麦语语音识别任务上验证了该方法可显著提升域内外泛化能力。

摘要:

尽管语音自监督编码器在海量且多样化的数据集上进行了训练,但在下游任务中通常仅被用作冻结的特征提取器或微调前的模型初始化器。前者严重限制了对大型编码器的利用,而后者则损害了预训练期间获得的鲁棒性,尤其是在低资源场景下。本文探索了中间方案,推测在下游微调期间减少对自监督任务的遗忘能够带来更好的泛化能力。为验证这一假设,本文聚焦于语音识别任务,在微调过程中对不同的持续学习方法进行了基准测试,并表明这些方法同时提升了域内和域外的泛化能力。在使用XLSR作为编码器的两个英语和丹麦语语音识别任务上,相对性能提升分别达到15.7%和22.5%。进一步的探测实验表明,这些性能提升确实与更少的遗忘相关。

英文摘要:

Despite being trained on massive and diverse datasets, speech self-supervised encoders are generally used for downstream purposes as mere frozen feature extractors or model initializers before fine-tuning. The former severely limits the exploitation of large encoders, while the latter hurts the robustness acquired during pretraining, especially in low-resource scenarios. This work explores middle-ground solutions, conjecturing that reducing the forgetting of the self-supervised task during the downstream fine-tuning leads to better generalization. To prove this, focusing on speech recognition, we benchmark different continual-learning approaches during fine-tuning and show that they improve both in-domain and out-of-domain generalization abilities. Relative performance gains reach 15.7% and 22.5% with XLSR used as the encoder on two English and Danish speech recognition tasks. Further probing experiments show that these gains are indeed linked to less forgetting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑