HEAL：一种基于假设的偏好感知分析框架
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
AI总结：
本文提出HEAL框架，将偏好对齐评估形式化为假设空间内的重排序，并构建UniHypoBench基准，通过排序准确率与偏好强度相关性两个指标揭示当前偏好学习方法能有效捕获代理模型偏好并抑制负样本，为对齐算法改进提供诊断工具。
摘要：
诸如DPO之类的偏好优化方法在LLM对齐中取得了显著性能。然而，对这些方法的评估依赖于单一响应，并忽略了其他潜在输出，而这些输出在现实世界应用中也可能在该假设空间内被生成。为解决这一问题，本文提出了基于假设的偏好感知分析框架（HEAL），这是一种新颖的评估范式，将偏好对齐形式化为假设空间内的重排序过程。该框架包含两个互补的度量指标：用于评估序数一致性的排序准确率，以及用于评估连续对齐程度的偏好强度相关性。为支持该框架，我们开发了UniHypoBench，一个由多样化的指令-响应对构建的统一假设基准。通过基于HEAL的大量实验，并特别关注偏好学习的内在机制，我们证明当前的偏好学习方法能够有效捕获由代理模型提供的偏好，同时抑制负样本。这些发现通过两条重要途径为偏好学习研究做出贡献。在理论上，我们引入假设空间分析作为一种理解偏好对齐的创新范式。在实践中，HEAL为研究人员提供了稳健的诊断工具，用于改进偏好优化方法，同时我们的实证结果指出了开发能够全面捕获偏好的更先进对齐算法的有前景方向。
英文摘要：
Preference optimization methods like DPO have achieved remarkable performance in LLM alignment. However, the evaluation for these methods relies on a single response and overlooks other potential outputs, which could also be generated in real-world applications within this hypothetical space. To address this issue, this paper presents a \textbf{H}ypothesis-based Pr\textbf{E}ference-aware \textbf{A}na\textbf{L}ysis Framework (HEAL), a novel evaluation paradigm that formulates preference alignment as a re-ranking process within hypothesis spaces. The framework incorporates two complementary metrics: ranking accuracy for evaluating ordinal consistency and preference strength correlation for assessing continuous alignment. To facilitate this framework, we develop UniHypoBench, a unified hypothesis benchmark constructed from diverse instruction-response pairs. Through extensive experiments based on HEAL, with a particular focus on the intrinsic mechanisms of preference learning, we demonstrate that current preference learning methods can effectively capture preferences provided by proxy models while simultaneously suppressing negative samples. These findings contribute to preference learning research through two significant avenues. Theoretically, we introduce hypothesis space analysis as an innovative paradigm for understanding preference alignment. Practically, HEAL offers researchers robust diagnostic tools for refining preference optimization methods, while our empirical results identify promising directions for developing more advanced alignment algorithms capable of comprehensive preference capture.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
