当大语言模型评分膨胀时：探索相关性评估中的过度评分
When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment
AI总结：
研究揭示LLM在相关性评估中存在系统性高估问题，指出模型对不满足信息需求的文本给出高分，且易受文本长度和表层词汇影响，强调需谨慎评估LLM在相关性评估中的应用。
摘要：
人类的相关性评估耗时且认知负担重，限制了信息检索评估的可扩展性。为此，人们日益关注使用大语言模型（LLMs）作为人类评估者的替代品。然而，LLM生成的相关性判断是否可靠、稳定且严谨仍存疑问。本文研究了不同模型架构、评估范式（点对点和成对）及段落修改策略下LLM的相关性判断中的过度评分行为。结果显示，模型倾向于对不真正满足信息需求的段落赋予高分，且信心较高，揭示出系统性偏见而非随机波动。进一步的受控实验表明，LLM的相关性判断对段落长度和表层词汇特征高度敏感。这些结果引发了对LLM作为人类评估者替代品的担忧，并突显了在应用LLM进行相关性评估时，需建立严谨的诊断评估框架的紧迫性。我们的代码和结果已公开。
英文摘要：
Human relevance assessment is time-consuming and cognitively intensive, limiting the scalability of Information Retrieval evaluation. This has led to growing interest in using large language models (LLMs) as proxies for human judges. However, it remains an open question whether LLM-based relevance judgments are reliable, stable, and rigorous enough to match humans for relevance assessment. In this work, we conduct a study of \textit{overrating behavior} in LLM-based relevance judgments across model backbones, evaluation paradigms (pointwise and pairwise), and passage modification strategies. We show that models consistently assign inflated relevance scores -- often with high confidence -- to passages that do not genuinely satisfy the underlying information need, revealing a system-wide bias rather than random fluctuations in judgment. Furthermore, controlled experiments show that LLM-based relevance judgments can be highly sensitive to passage length and surface-level lexical cues. These results raise concerns about the usage of LLMs as drop-in replacements for human relevance assessors, and highlight the urgent need for careful diagnostic evaluation frameworks when applying LLMs for relevance assessments. Our code and results are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
