克服大规模多语言持续学习中的灾难性遗忘
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
AI总结：
本文针对多达51种语言的大规模多语言持续学习中的灾难性遗忘问题，提出学习率调度方法LR ADJUST，在分类和序列标注任务上验证其能在保留新知识的同时减少对旧知识的覆盖，并分析了遗忘动态。
摘要：
现实生活中的多语言系统应当能够随着输入系统的数据分布随时间演变和变化，高效地纳入新语言。为此，系统需要处理灾难性遗忘问题，即模型在更早见过的语言或任务上性能下降。在本文中，我们在一个涉及多达51种语言、同时覆盖分类和序列标注任务的大规模多语言持续学习框架下，研究灾难性遗忘以及缓解该问题的方法。我们提出了LR ADJUST，这是一种学习率调度方法，简单但在保留新信息的同时有效避免强烈覆盖过去的知识。此外，我们表明该方法在多种持续学习方法中均有效。最后，我们进一步揭示了在这种大规模多语言设置下灾难性遗忘的动态特性。
英文摘要：
Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
