AIPO：改进迭代偏好优化的训练目标
AIPO: Improving Training Objective for Iterative Preference Optimization
AI总结：
针对迭代偏好优化中更严重的长度利用问题，提出协议感知迭代偏好优化（AIPO）训练目标，在多个基准上取得SOTA性能。
摘要：
偏好优化（Preference Optimization, PO）作为对齐大语言模型（Large Language Models, LLMs）的近端策略优化（Proximal Policy Optimization, PPO）替代方案，正日益受到关注。近期使用合成或部分合成数据迭代对齐LLMs的研究，在学术场景及Llama3等专有训练模型的PO训练规模化方面展现出可观成果。尽管取得了成功，我们的研究表明，由于过程的迭代特性，PO中存在的长度利用问题在迭代偏好优化（Iterative Preference Optimization, IPO）中更为严重。本研究针对使用合成数据的迭代偏好优化展开，分享了构建迭代偏好优化流水线过程中的发现与分析。具体而言，我们探讨了迭代偏好优化期间的长度利用问题，并提出了面向迭代偏好优化的训练目标，即协议感知迭代偏好优化（Agreement-aware Iterative Preference Optimization, AIPO）。为验证方法的有效性，我们开展了全面实验，在MT-Bench、AlpacaEval 2.0和Arena-Hard上取得了当前最优性能。我们的实现代码与模型检查点将发布于https://github.com/bytedance/AIPO。
英文摘要：
Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at https://github.com/bytedance/AIPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
