通过偏好之上的偏好进行自适应边际基于人类反馈的强化学习
Adaptive Margin RLHF via Preference over Preferences
AI总结：
研究在基于人类反馈的强化学习中奖励模型学习的边际优化问题，提出利用偏好之上的偏好推断自适应边际，介绍具体实例DPO-PoP，提升判别和生成性能，并给出采样策略。
摘要：
基于边际的优化对于提高分类任务的泛化和鲁棒性至关重要。在基于人类反馈的强化学习中奖励模型从偏好学习的背景下，现有方法存在不足。我们提出利用偏好之上的偏好来推断每个数据点的自适应边际，引入DPO-PoP扩展，展示了判别和生成性能之间的权衡并提出采样策略。
英文摘要：
Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing methods typically rely on no margins, fixed margins, or margins that are simplistic functions of preference ratings. However, such formulations often fail to account for the varying strengths of different preferences or they rely on noisy margin information derived from preference ratings. Furthermore, many existing methods that use adaptive margins assume access to accurate preference scores, which can be difficult for humans to provide reliably. We propose leveraging preferences over preferences, that is, annotations indicating which of two preferences reflects a stronger distinction, to infer adaptive margins on a per-datapoint basis. Such preference-over-preference annotations are general and can be incorporated into both standard RLHF reward modeling objectives and direct alignment losses. As a concrete instantiation, we introduce DPO-PoP, an extension to Direct Preference Optimization (DPO) that incorporates adaptive margins from preference-over-preference supervision, enabling improved discriminative and generative performance. Additionally, we show a tradeoff between discriminative and generative performance and propose two sampling strategies for gathering preference-over-preference labels to navigate it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
