通过项目反应理论诊断LLM作为评判者的可靠性
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
AI总结：
提出基于项目反应理论（IRT）的两阶段诊断框架，通过内在一致性和人类对齐两个维度评估LLM作为评判者的可靠性，并提供可解释的诊断信号。
摘要：
虽然LLM作为评判者（LLM-as-a-Judge）在自动评估中被广泛使用，但现有的验证实践主要在观察输出层面进行，对于LLM评判者本身是否作为稳定可靠的测量工具提供的洞察有限。为了解决这一局限性，我们引入了一个基于项目反应理论（IRT）的两阶段诊断框架，用于评估LLM作为评判者的可靠性。该框架采用IRT的分级响应模型（GRM），并沿两个互补维度形式化可靠性：（1）内在一致性，定义为在提示变化下测量行为的稳定性，以及（2）人类对齐，捕捉与人类质量评估的一致性。我们通过该框架实证检验了多种LLM评判者，并表明利用IRT-GRM可以产生可解释的信号，用于系统性地诊断判断。这些信号为验证LLM作为评判者的可靠性以及识别不可靠性的潜在原因提供了实用指导。
英文摘要：
While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
