LLM微调的学习动态
Learning Dynamics of LLM Finetuning
AI总结：
本文通过分析影响累积的逐步分解研究了LLM微调的学习动态，提出了统一解释框架，揭示了幻觉强化和离策略DPO“挤压效应”的成因，并启发了一种提升对齐性能的简单方法。
摘要：
学习动态描述了对特定训练样本的学习如何影响模型对其他样本的预测，为我们理解深度学习系统的行为提供了强大工具。我们通过分析不同潜在响应间影响累积的逐步分解，研究了大型语言模型在不同类型微调期间的学习动态。我们的框架对关于指令微调和偏好微调的流行算法训练的许多有趣观察提供了一致解释。特别地，我们提出了一种假设性解释，说明为何特定类型的幻觉在微调后被强化，例如，模型可能会使用问题B的响应中的短语或事实来回答问题A，或者模型在生成响应时可能会不断重复相似的简单短语。我们还扩展了框架，并强调了一种独特的“挤压效应”，以解释在离策略直接偏好优化（DPO）中先前观察到的现象，即运行DPO过久会使即使期望的输出也变得可能性更低。该框架还提供了关于在策略DPO及其他变体的益处来源的见解。该分析不仅提供了理解LLM微调的新视角，还启发了一种简单有效的方法来提升对齐性能。
英文摘要：
Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
