克服LLM微调中的遗忘：进化策略方法
Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
AI总结：
本文发现进化策略微调中的先前任务遗忘实为性能漂移且可恢复，并引入锚定权重衰减（AWD）正则化技术有效稳定先前任务性能，表明遗忘可避免，使ES成为LLM持续学习的可行方法。
摘要：
进化策略（ES）最近作为强化学习（RL）在大语言模型（LLM）微调中的竞争性替代方案出现，通过简单性、可扩展性和仅推理训练提供优势。然而，近期研究表明，在新任务上进行ES微调可能导致对先前任务的遗忘。首先，本文表明先前任务遗忘（1）更好地被描述为性能漂移而非不可逆遗忘，在ES训练过程中先前任务性能通常会恢复；（2）并非ES特有的失败模式，使用RL方法微调时也可能出现。其次，本文分析了这种漂移何时以及为何出现，强调了其对ES训练动态的依赖性，特别是权重空间中弱约束方向上的随机游走行为。第三，基于这些见解，本文引入了锚定权重衰减（AWD）作为一种参数空间正则化技术，将优化约束向初始模型参数。AWD在保持目标任务性能的同时有效稳定了先前任务性能，以更低的计算成本实现了与大型ES种群规模相当的优势。因此，与先前观点相反，本文表明ES下的先前任务遗忘在很大程度上是可以避免的，使ES成为LLM持续学习中一种有前景的方法。
英文摘要：
Evolution Strategies (ES) has recently emerged as a competitive alternative to reinforcement learning (RL) for large language model (LLM) fine-tuning, offering advantages through simplicity, scalability, and inference-only training. However, recent work suggests that ES fine-tuning on new tasks may induce forgetting of prior tasks. First, this paper shows that prior task forgetting (1) is better characterized as performance drift rather than irreversible forgetting, with prior-task performance often recovering during ES training; and (2) is not a specific failure mode of ES, but can also arise for fine-tuning with RL methods. Second, it analyzes when and why such drift arises, highlighting its dependence on ES training dynamics, particularly random walk behavior in weakly constrained directions of the weight space. Third, based on these insights, it introduces Anchored Weight Decay (AWD) as a parameter-space regularization technique that constrains optimization toward the initial model parameters. AWD effectively stabilizes prior-task performance while preserving target-task performance, achieving benefits comparable to large ES population sizes at much lower computational cost. Thus, contrary to previous beliefs, the paper shows that prior-task forgetting under ES is largely avoidable, positioning ES as a promising approach for continual learning in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
