arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.02219cs.CL
收藏

我是更偏向逐点还是成对?揭示基于评分标准的LLM作为评判者的位置偏差

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

OMRON SINIC X;NexaScience

Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, Yoshitaka Ushiku

AI总结:

研究基于评分标准的LLM评估中的位置偏差,发现其类似多项选择,模型倾向于选择特定位置的选项,偏差方向因模型而异,且评分标准顺序也会影响结果。

摘要:

大型语言模型被广泛用作评估者,这种范式通常称为LLM-as-a-judge。先前的研究主要考察了点式或成对评估协议;相比之下,我们关注的是基于评分标准的评估,由于其在验证困难的领域中用于训练模型而日益受到关注。在这项工作中,我们表明基于评分标准的评估隐式地类似于多项选择设置,因此表现出位置偏差:LLM倾向于选择评分标准列表中特定位置出现的分数选项。通过在多个模型和数据集上的控制实验,我们证明这种位置偏差是一致的。然而,其方向是模型特定的:一些评判者偏爱第一个选项,而另一些则偏爱最后一个。我们进一步识别出第二个正交的偏差轴:当提示同时评分多个标准时,标准本身的顺序会改变最终得分。我们还探索了通过排列评分标准选项的顺序来减轻位置偏差,并发现虽然偏差可以被减弱,但模型判断与人工注释之间相关性的改善主要出现在表现出强偏差的模型中。我们的结果将基于评分标准的LLM-as-a-judge重新定义为具有可测量、模型特定位置偏差的多项选择问题,并进一步证实,对于大多数模型,只需少量随机顺序排列即可减少由该偏差引入的误差。

英文摘要:

Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examined point-wise or pair-wise evaluation protocols; in contrast, our focus is on rubric-based evaluation, which has been attracting increasing attention owing to its utility for training models in domains where verification is otherwise difficult. In this work, we show that rubric-based evaluation implicitly resembles a multiple-choice setting and therefore exhibits position bias: LLMs tend to prefer score options that appear at specific positions within the rubric list. Through controlled experiments across multiple models and datasets, we demonstrate that this position bias is consistent. Its direction, however, is model-specific: some judges favor the first option, while others favor the last. We further identify a second, orthogonal axis of bias: when a prompt scores several criteria simultaneously, the ordering of the criteria itself shifts the resulting scores. We additionally explore permuting the order of the rubric options as a means of mitigating position bias, and find that although the bias can be attenuated, improvements in the correlation between model judgments and human annotations are obtained primarily for models that exhibit strong bias. Our results recast rubric-based LLM-as-a-judge as a multiple-choice problem with measurable, model-specific position bias, and we further confirm that only a small number of random order permutations are sufficient to reduce the error introduced by this bias for the majority of models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑