ParamΔ 用于直接权重混合：零成本的后训练大语言模型
Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost
AI总结：
ParamΔ通过零成本方法实现基础模型的后训练能力迁移，提升模型性能与开发效率。
摘要：
大语言模型的后训练阶段对于增强诸如遵循指令、推理和与人类偏好对齐等能力至关重要。然而，这一过程需要大量高质量的数据，并存在过拟合的风险，同时每次基础模型更新后都需要重复的后训练和评估，导致显著的计算成本。本文介绍了一种新的方法ParamΔ，通过将现有后训练模型的知识转移到新更新的基础模型中，无需额外训练即可简化后训练过程。通过计算后训练模型权重（Θ_post）与基础模型权重（Θ_base）之间的差异，并将其加到更新后的基础模型（Θ'_base）上，我们定义ParamΔ模型为：Θ_ParamΔ = Θ_post - Θ_base + Θ'_base。这种方法意外地使新基础模型具备了后训练的能力，其性能与直接后训练相当。我们对LLama3、LLama3.1、Qwen和DeepSeek-distilled模型进行了分析。结果显示ParamΔ模型有效复制了传统后训练。例如，从70B Llama3-inst、Llama3-base、Llama3.1-base模型获得的ParamΔ模型在平均上达到了Llama3.1-inst模型性能的约95%。ParamΔ为开放权重社区中如何充分利用模型提供了一种新的视角，其中基础和指导模型的检查点 readily available且频繁更新，通过提供一个无需成本的框架来加速模型开发的迭代周期。
英文摘要：
The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it demands extensive high-quality data and poses risks like overfitting, alongside significant computational costs due to repeated post-training and evaluation after each base model update. This paper introduces $ParamΔ$, a novel method that streamlines post-training by transferring knowledge from an existing post-trained model to a newly updated base model with ZERO additional training. By computing the difference between post-trained model weights ($Θ_\text{post}$) and base model weights ($Θ_\text{base}$), and adding this to the updated base model ($Θ'_\text{base}$), we define $ParamΔ$ Model as: $Θ_{\text{Param}Δ} = Θ_\text{post} - Θ_\text{base} + Θ'_\text{base}$. This approach surprisingly equips the new base model with post-trained capabilities, achieving performance comparable to direct post-training. We did analysis on LLama3, Llama3.1, Qwen, and DeepSeek-distilled models. Results indicate $ParamΔ$ Model effectively replicates traditional post-training. For example, the $ParamΔ$ Model obtained from 70B Llama3-inst, Llama3-base, Llama3.1-base models attains approximately 95\% of Llama3.1-inst model's performance on average. $ParamΔ$ brings a new perspective on how to fully leverage models in the open-weight community, where checkpoints for base and instruct models are readily available and frequently updated, by providing a cost-free framework to accelerate the iterative cycle of model development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
