可证明地避免在不了解数据分布的情况下直接偏好优化中的过度优化
Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution
AI总结：
PEPO通过训练不同数据子集的偏好优化策略并聚合最坏情况，避免了过度优化问题，其样本复杂度仅依赖单策略集中性系数，保持了DPO的简洁性和实用性。
摘要：
我们引入PEPO（基于悲观的偏好优化），一种单步直接偏好优化（DPO）类算法，用于缓解偏好学习中已知的过度优化问题，而无需了解数据生成分布或学习显式奖励模型。PEPO通过训练不同数据子集的偏好优化策略的集合，并通过最坏情况构造聚合，以促进模型间的共识。在表格设置中，PEPO的样本复杂度仅依赖于单策略集中性系数，从而避免了影响易产生过度优化算法（如DPO）保证的所有策略集中性。理论发现通过令人信服的实践性能得到验证，同时保留了DPO风格训练的简洁性和实用性。
英文摘要：
We introduce PEPO (Pessimistic Ensemble based Preference Optimization), a single-step Direct Preference Optimization (DPO)-like algorithm to mitigate the well-known over-optimization issue in preference learning without requiring the knowledge of the data-generating distribution or learning an explicit reward model. PEPO achieves pessimism via an ensemble of preference-optimized policies trained on disjoint data subsets and then aggregates them through a worst case construction that favors the agreement across models. In the tabular setting, PEPO achieves sample complexity guarantees depending only on a single-policy concentrability coefficient, thus avoiding the all-policy concentrability which affects the guarantees of algorithms prone to over-optimization, such as DPO. The theoretical findings are corroborated by a convincing practical performance, while retaining the simplicity and the practicality of DPO-style training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
