Chatbot Arena 遇见 Nuggets：面向 LLM 响应评估的解释与诊断
Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
AI总结：
针对 LLM 响应评估中现有对战方法缺乏解释性和诊断性的问题，本文应用 AutoNuggetizer 框架自动分析约 7K 个 Search Arena 对战数据，发现 nugget 分数与人类偏好显著相关，为可解释和可诊断的系统评估提供了新途径。
摘要：
对战，即在所谓的竞技场中并排比较以引发人类偏好的方式，已成为评估 LLM 输出质量的一种流行方法。最近，这一思想已被扩展到检索增强生成（RAG）系统。虽然对战无疑代表了评估方面的进步，但它至少存在两个缺点，尤其是在复杂的信息寻求查询背景下：它们既不具有解释性，也不具有诊断性。近期，nugget 评估方法作为一种评估 RAG 答案质量的有前景的方法而出现。Nuggets 将长形式的 LLM 生成答案分解为原子事实，突出“好”回答中必要的重要信息片段。在本工作中，我们应用我们的 AutoNuggetizer 框架，以完全自动化的方式分析了 LMArena 提供的约 7K 个 Search Arena 对战数据。我们的结果显示，nugget 分数与人类偏好之间存在显著相关性，展示了我们在可解释和可诊断系统评估方法上的潜力。复现我们工作结果所需的所有代码可在 https://github.com/castorini/lmsys_nuggetize 获取。
英文摘要：
Battles, or side-by-side comparisons in so-called arenas that elicit human preferences, have emerged as a popular approach for assessing the output quality of LLMs. Recently, this idea has been extended to retrieval-augmented generation (RAG) systems. While undoubtedly representing an advance in evaluation, battles have at least two drawbacks, particularly in the context of complex information-seeking queries: they are neither explanatory nor diagnostic. Recently, the nugget evaluation methodology has emerged as a promising approach to evaluate the quality of RAG answers. Nuggets decompose long-form LLM-generated answers into atomic facts, highlighting important pieces of information necessary in a "good" response. In this work, we apply our AutoNuggetizer framework to analyze data from roughly 7K Search Arena battles provided by LMArena in a fully automatic manner. Our results show a significant correlation between nugget scores and human preferences, showcasing promise in our approach to explainable and diagnostic system evaluations. All the code necessary to reproduce results in our work is available in https://github.com/castorini/lmsys_nuggetize.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
