arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09656cs.LGcs.CL
收藏

学习你的参考模型以实现真正的对齐

Learn Your Reference Model for Real Good Alignment

T-Tech

Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, Daniil Gavrilov

AI总结:

提出信任区域(TR)离线对齐方法,动态更新参考策略以缓解过度优化,在对话、摘要及通用助手任务中超越传统方法。

摘要:

尽管大语言模型(LLM)对齐的离线方法不需要直接奖励模型,但它们仍然容易受到过度优化的影响。当训练后的模型过度偏离参考策略时,就会出现这个问题,导致样本质量下降。我们提出了一种新的离线对齐方法范式,称为信任区域(包括变体TR-DPO、TR-IPO、TR-KTO),它在整个训练过程中动态更新参考策略。我们的结果表明,TR对齐方法有效缓解了过度优化,使模型即使在显著偏离初始参考策略时也能保持强劲性能。我们不仅通过展示减少过度优化的玩具示例证明了这些方法的有效性,还在特定任务中进行了直接的并排比较,如有用且无害的对话以及摘要生成,在这些任务中它们超越了传统方法。此外,我们报告了在AlpacaEval 2和Arena-Hard基准上使用Llama3模型在通用助手设置中的显著改进,突出了信任区域方法相对于经典方法的优势。

英文摘要:

Despite the fact that offline methods for Large Language Models (LLMs) alignment do not require a direct reward model, they remain susceptible to overoptimization. This issue arises when the trained model deviates excessively from the reference policy, leading to a decrease in sample quality. We propose a new paradigm of offline alignment methods, called Trust Region (including variants TR-DPO, TR-IPO, TR-KTO), which dynamically updates the reference policy throughout the training process. Our results show that TR alignment methods effectively mitigate overoptimization, enabling models to maintain strong performance even when substantially deviating from the initial reference policy. We demonstrate the efficacy of these approaches not only through toy examples that exhibit reduced overoptimization, but also through direct, side-by-side comparisons in specific tasks such as helpful and harmless dialogue, as well as summarization, where they surpass conventional methods. Additionally, we report significant improvements in general-purpose assistant setups with the Llama3 model on the AlpacaEval 2 and Arena-Hard benchmarks, highlighting the advantages of Trust Region methods over classical approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑