进化策略导致大语言模型中的灾难性遗忘
Evolutionary Strategies lead to Catastrophic Forgetting in LLMs
AI总结：
进化策略在大语言模型训练中导致灾难性遗忘，尽管性能接近GRPO，但持续学习能力受限。
摘要：
当前AI系统最大的缺失能力是部署后能够持续学习的能力。实现此类持续学习系统有几个挑战，其中之一是梯度基于算法在训练最先进的大语言模型（LLM）时所需的大量内存。进化策略（ES）最近重新出现作为传统学习算法的梯度自由替代方案，并在LLM特定任务上表现出有希望的性能。在本文中，我们对ES进行了全面分析，并特别评估了在增加更新步骤数时其遗忘曲线。我们首先发现，ES在数学和推理任务上可以达到与GRPO相当的性能水平，使用可比的计算预算。然而，对于持续学习来说，最重要的是，ES的性能提升伴随着显著的先前能力遗忘，限制了其在在线训练模型中的适用性。我们还探索了这种行为背后的原因，并表明使用ES进行的更新比相应的GRPO更新要稀疏得多，并且具有数量级更大的$\ell_2$范数，解释了两种算法之间遗忘曲线的对比。通过这项研究，我们旨在突出梯度自由算法如ES中的遗忘问题，并希望激发未来工作以缓解这些问题。
英文摘要：
One of the biggest missing capabilities in current AI systems is the ability to learn continuously after deployment. Implementing such continually learning systems have several challenges, one of which is the large memory requirement of gradient-based algorithms that are used to train state-of-the-art LLMs. Evolutionary Strategies (ES) have recently re-emerged as a gradient-free alternative to traditional learning algorithms and have shown encouraging performance on specific tasks in LLMs. In this paper, we perform a comprehensive analysis of ES and specifically evaluate its forgetting curves when training for an increasing number of update steps. We first find that ES is able to reach performance numbers close to GRPO for math and reasoning tasks with a comparable compute budget. However, and most importantly for continual learning, the performance gains in ES is accompanied by significant forgetting of prior abilities, limiting its applicability for training models online. We also explore the reason behind this behavior and show that the updates made using ES are much less sparse and have orders of magnitude larger $\ell_2$ norm compared to corresponding GRPO updates, explaining the contrasting forgetting curves between the two algorithms. With this study, we aim to highlight the issue of forgetting in gradient-free algorithms like ES and hope to inspire future work to mitigate these issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
