无人类触感:基于LLM的摘要评估中的重叠偏差
Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
AI总结:
本研究发现LLM在摘要评估中对重叠度敏感,随着重叠减少,LLM更倾向于选择其他LLM生成的摘要,表明需采用更复杂的评判技术。
摘要:
大型语言模型(LLM)评判者常与传统算法指标一起用于摘要任务,因为它们能更好地捕捉语义信息,推理能力更强,且对改写更稳健。然而,LLM评判者在长度和顺序等方面表现出偏见,并且容易受到各种对抗性输入提示的影响。尽管最近的研究已探讨了这些偏见,但很少有人在更细粒度的层面,结合明确的重叠指标进行分析。在本工作中,我们提供了一种LLM评判偏见分析,该分析基于与人类撰写响应的重叠情况。我们测试了9种最近的LLM,参数量从10亿到120亿不等,包括Gemma 3和LLaMA 3的变种。我们发现,随着被评判摘要与人类生成摘要之间的相似性(通过ROUGE和BLEU测量)降低,LLM评判者越来越倾向于选择由其他LLM生成的摘要而非人类生成的摘要,且这一模式适用于所有但一个测试模型,并且与模型自身的立场偏见无关。此外,我们发现模型难以评判即使具有有限重叠的摘要,这表明在摘要领域,LLM作为评判者应依赖超越简单比较的技术。
英文摘要:
Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization because they better capture semantic information, are better at reasoning, and are more robust to paraphrasing. However, LLM judges show biases for length and order among others, and are vulnerable to various adversarial input prompts. While recent studies have looked into these biases, few have analyzed them at a more granular level in relation to a well-defined overlap metric. In this work we provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization. We test 9 recent LLMs with parameter counts ranging from 1 billion to 12 billion, including variants of Gemma 3 and LLaMA 3. We find that LLM judges increasingly prefer summaries generated by other LLMs over those written by humans as the similarities (as measured by ROUGE and BLEU) between the judged summaries decrease, and this pattern extends to all but one model tested, and exists regardless of the models' own position biases. Additionally, we find that models struggle to judge even summaries with limited overlaps, suggesting that LLM-as-a-judge in the summary domain should rely on techniques beyond a simple comparison.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。