RocketEval：通过评分清单实现高效自动化大语言模型评估
RocketEval: Efficient Automated LLM Evaluation via Grading Checklist
AI总结：
RocketEval提出用轻量级LLM结合实例特定清单评分实现高效自动化评估，在MT-Bench和WildBench上以Gemma-2-2B达到与GPT-4o相当的人类偏好相关性，并降低超50倍成本。
摘要：
在多样且具有挑战性的场景中评估大语言模型（LLMs）对于使其与人类偏好对齐至关重要。为了降低与人工评估相关的高昂成本，使用强大的大语言模型作为评判者已成为一种备受青睐的方法。然而，这种方法面临若干挑战，包括巨大的开销、隐私与安全问题以及可复现性。在本文中，我们提出了一种简单、可复现且准确的自动化评估方法，通过利用轻量级大语言模型作为评判者，命名为RocketEval。首先，我们发现轻量级与强大LLMs在评估任务中的性能差距主要源于它们进行全面分析的能力，而这种能力并不容易通过思维链推理等技术得到提升。通过将评估任务重新构建为基于实例特定清单的多方面问答，我们证明轻量级LLMs有限的判断准确性在很大程度上归因于高不确定性和位置偏差。为应对这些挑战，我们引入了一种基于清单评分的自动化评估流程，旨在适应各种场景和问题。该流程包括清单的创建、由轻量级LLMs对这些清单进行评分，以及对清单项目进行重新加权以与监督标注对齐。我们在自动化评估基准MT-Bench和WildBench数据集上进行的实验表明，RocketEval在使用Gemma-2-2B作为评判者时，与人类偏好实现了高达0.965的相关性，与GPT-4o相当。此外，RocketEval在大规模评估和比较场景中提供了超过50倍的成本降低。我们的代码可在https://github.com/Joinn99/RocketEval-ICLR获取。
英文摘要：
Evaluating large language models (LLMs) in diverse and challenging scenarios is essential to align them with human preferences. To mitigate the prohibitive costs associated with human evaluations, utilizing a powerful LLM as a judge has emerged as a favored approach. Nevertheless, this methodology encounters several challenges, including substantial expenses, concerns regarding privacy and security, and reproducibility. In this paper, we propose a straightforward, replicable, and accurate automated evaluation method by leveraging a lightweight LLM as the judge, named RocketEval. Initially, we identify that the performance disparity between lightweight and powerful LLMs in evaluation tasks primarily stems from their ability to conduct comprehensive analyses, which is not easily enhanced through techniques such as chain-of-thought reasoning. By reframing the evaluation task as a multi-faceted Q&A using an instance-specific checklist, we demonstrate that the limited judgment accuracy of lightweight LLMs is largely attributes to high uncertainty and positional bias. To address these challenges, we introduce an automated evaluation process grounded in checklist grading, which is designed to accommodate a variety of scenarios and questions. This process encompasses the creation of checklists, the grading of these checklists by lightweight LLMs, and the reweighting of checklist items to align with the supervised annotations. Our experiments carried out on the automated evaluation benchmarks, MT-Bench and WildBench datasets, reveal that RocketEval, when using Gemma-2-2B as the judge, achieves a high correlation (0.965) with human preferences, which is comparable to GPT-4o. Moreover, RocketEval provides a cost reduction exceeding 50-fold for large-scale evaluation and comparison scenarios. Our code is available at https://github.com/Joinn99/RocketEval-ICLR .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
