模型外推加速对齐
Model Extrapolation Expedites Alignment
AI总结:
针对大语言模型偏好对齐训练成本高的问题,提出基于一阶近似的模型外推方法 ExPO,通过放大参数变化加速对齐,无额外开销且显著提升模型性能。
摘要:
鉴于大语言模型(LLMs)偏好对齐训练的高计算成本,探索降低训练开销的高效方法仍是一个重要且引人关注的研究问题。受对齐训练通常仅涉及参数微小变化而不向模型注入新知识这一观察的启发,我们提出了一种名为 ExPO(model extrapolation,模型外推)的简单方法,以加速 LLMs 与人类偏好的对齐。给定一个部分训练的模型及其初始 SFT 检查点,ExPO 基于一阶近似,仅通过放大参数变化来改进对齐训练的隐式优化目标,且无任何额外训练开销。通过对照实验,我们证明 ExPO 能将仅用 20% 步数训练的 DPO 模型提升至超越完全训练的模型。此外,我们表明 ExPO 显著提升了现有开源 LLMs(参数规模从 1.8B 到 70B)在领先的 AlpacaEval 2.0 和 MT-Bench 基准上的表现,凸显了 ExPO 在高效增强 LLM 对齐方面的更广泛实用性。
英文摘要:
Given the high computational cost of preference alignment training of large language models (LLMs), exploring efficient methods to reduce the training overhead remains an important and compelling research problem. Motivated by the observation that alignment training typically involves only small parameter changes without injecting new knowledge into models, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs' alignment with human preferences. Given a partially-trained model and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, without any additional training overhead. Through controlled experiments, we demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one. Moreover, we show that ExPO notably improves existing open-source LLMs (ranging from 1.8B to 70B parameters) on the leading AlpacaEval 2.0 and MT-Bench benchmarks, which highlights ExPO's broader utility in efficiently enhancing LLM alignment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。