arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12660cs.CL
收藏

从偏好中学习序概率奖励

Learning Ordinal Probabilistic Reward from Preferences

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Ritzz-AI;Shenzhen University of Advanced Technology(深圳先进技术大学)

Longze Chen, Qi Deng, Lu Wang, Renke Shan, Ze Gong, Run Luo, Jiaming Li, Jing Luo, Qiyao Wang, Min Yang

AI总结:

本文提出序概率奖励模型,通过区域淹没微调提升奖励模型的准确性和数据效率,有效捕捉绝对文本质量。

摘要:

奖励模型对于对齐大语言模型(LLMs)与人类价值观和意图至关重要。现有方法遵循生成(GRMs)或判别(DRMs)范式,但两者都存在局限性:GRMs通常需要昂贵的点对点监督,而DRMs产生未经校准的相对分数,缺乏概率解释。为了解决这些挑战,我们介绍了一种新的奖励建模范式:概率奖励模型(PRM)。与将奖励视为确定性标量不同,我们的方法将奖励视为随机变量,学习每个响应质量的完整概率分布。为了使该范式实用,我们提出了其闭式、离散实现:序概率奖励模型(OPRM),将质量分数离散化为有限的序评级集。基于OPRM,我们提出了一种数据高效的训练策略,称为区域淹没微调(RgFT)。它使奖励能够更好地反映绝对文本质量,通过整合质量级别注释,引导模型将概率质量集中在相应的评级子区域中。在各种奖励模型基准测试中的实验表明,我们的方法在准确性上比先前的奖励模型提高了2.9%~7.4%,展示了强大的性能和数据效率。对分数分布的分析提供了证据,表明我们的方法不仅捕捉了相对排名,还捕捉了绝对质量。

英文摘要:

Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that our method improves accuracy by $\textbf{2.9%}\sim\textbf{7.4%}$ compared to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑