F-MALLOC：神经机器翻译持续学习中的前馈记忆分配
F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation
AI总结：
F-MALLOC通过将前馈层分解为离散记忆单元并按任务分配，有效缓解神经机器翻译持续学习中的灾难性遗忘，同时保持系统可扩展性，实验显示更高BLEU分数和几乎零遗忘。
摘要：
在神经机器翻译（NMT）不断发展的背景下，预训练-微调范式已取得显著成果，但灾难性遗忘（CF）这一持续存在的挑战仍然是障碍。此前工作引入了持续学习（CL）方法来应对CF，但这些方法在避免遗忘与保持系统可扩展性之间难以取得微妙平衡。为此，我们提出一种名为F-MALLOC（前馈记忆分配）的持续学习方法。F-MALLOC受近期研究启发，这些研究指出前馈层模拟神经记忆并封装关键的翻译知识。该方法将前馈层分解为离散的记忆单元，并将这些记忆分配给不同任务。通过学习分配和保护这些记忆，我们的方法有效缓解了CF，同时确保了稳健的可扩展性。此外，我们提出了一套针对NMT系统多阶段持续学习的综合评估协议。按照这一新协议进行的实验展示了F-MALLOC的优越性能，表现为更高的BLEU分数和几乎为零的遗忘。
英文摘要：
In the evolving landscape of Neural Machine Translation (NMT), the pretrain-then-finetune paradigm has yielded impressive results. However, the persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While previous work has introduced Continual Learning (CL) methods to address CF, these approaches grapple with the delicate balance between avoiding forgetting and maintaining system extensibility. To address this, we propose a CL method, named $\textbf{F-MALLOC}$ ($\textbf{F}$eed-forward $\textbf{M}$emory $\textbf{ALLOC}ation)$. F-MALLOC is inspired by recent insights highlighting that feed-forward layers emulate neural memories and encapsulate crucial translation knowledge. It decomposes feed-forward layers into discrete memory cells and allocates these memories to different tasks. By learning to allocate and safeguard these memories, our method effectively alleviates CF while ensuring robust extendability. Besides, we propose a comprehensive assessment protocol for multi-stage CL of NMT systems. Experiments conducted following this new protocol showcase the superior performance of F-MALLOC, evidenced by higher BLEU scores and almost zero forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
