置信评判:将自动评分器校准至偏好分布
Judging with Confidence: Calibrating Autoraters to Preference Distributions
AI总结:
针对大语言模型自动评分器因离散标签训练难以适配主观任务的问题,提出校准至偏好分布的通用框架及两种适配不同数据的学习方法,可提升对齐度与校准效果、降低位置偏差且保留客观任务性能。
摘要:
大语言模型(LLM)与人类价值观的对齐愈发依赖将其他大语言模型用作自动评判器,即“自动评分器(autoraters)”。但其可靠性受一个根本问题限制:它们基于离散偏好标签训练,会给通常具有主观性、模糊性或细微差别的任务强行设定唯一标准答案。我们认为,可靠的自动评分器必须学会建模目标群体定义的完整偏好分布。本文提出一个通用框架,用于将概率型自动评分器校准至任意给定的偏好分布。我们对该问题进行了形式化定义,并提出两种适配不同数据条件的学习方法:1)针对稠密概率标签的直接监督微调;2)针对稀疏二值标签的强化学习方法。实验结果表明,以分布匹配为目标微调自动评分器,可使其输出的概率预测与目标偏好分布更对齐,校准效果提升,位置偏差显著降低,同时还能保持在客观任务上的性能。
英文摘要:
The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。