语境重要吗？用于评估语境场景下基于大语言模型的评判器的ContextualJudgeBench基准
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
AI总结：
针对现有LLM评判器缺乏语境场景评估的问题，提出包含2000个响应对的ContextualJudgeBench基准，实验发现即使SOTA模型在该基准上的一致准确率也仅约55%。
摘要：
大语言模型（LLM）作为评判器的范式已被用于满足AI系统开发和部署后监控期间对模型输出进行低成本、可靠且快速评估的需求。尽管评判模型——即经过微调、专门负责评估和 critique 模型输出的大语言模型——被吹捧为通用评估工具，但它们通常仅在非语境场景下接受评估，例如指令遵循任务。鉴于检索增强生成（RAG）和摘要用例的日益普及，语境场景（即利用外部信息作为语境生成输出的场景）的缺失令人意外。语境评估具有独特的挑战性，因为评估往往取决于从业者的优先级，从而产生条件性评估标准（例如，先基于事实性比较响应，如果事实性相同则再考虑完整性）。为填补这一空白，我们提出了ContextualJudgeBench，这是一个评判器基准，包含受现实世界语境评估场景启发的8个拆分集下的2000个具有挑战性的响应对。我们通过多 pronged 数据构建流水线构建该基准，同时利用现有的人工标注和基于模型的扰动方法。我们对11个评判模型和9个通用模型开展的综合研究表明，语境信息及其评估标准即使对最先进的模型也构成了重大挑战。例如，表现最佳的模型OpenAI的o1，其一致准确率仅勉强达到55%。
英文摘要：
The large language model (LLM)-as-judge paradigm has been used to meet the demand for a cheap, reliable, and fast evaluation of model outputs during AI system development and post-deployment monitoring. While judge models -- LLMs finetuned to specialize in assessing and critiquing model outputs -- have been touted as general purpose evaluators, they are typically evaluated only on non-contextual scenarios, such as instruction following. The omission of contextual settings -- those where external information is used as context to generate an output -- is surprising given the increasing prevalence of retrieval-augmented generation (RAG) and summarization use cases. Contextual assessment is uniquely challenging, as evaluation often depends on practitioner priorities, leading to conditional evaluation criteria (e.g., comparing responses based on factuality and then considering completeness if they are equally factual). To address the gap, we propose ContextualJudgeBench, a judge benchmark with 2,000 challenging response pairs across eight splits inspired by real-world contextual evaluation scenarios. We build our benchmark with a multi-pronged data construction pipeline that leverages both existing human annotations and model-based perturbations. Our comprehensive study across 11 judge models and 9 general purpose models, reveals that the contextual information and its assessment criteria present a significant challenge to even state-of-the-art models. For example, OpenAI's o1, the best-performing model, barely reaches 55% consistent accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
