指令微调大语言模型的自动评估方法有多可靠？
How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?
AI总结：
本文通过元评估并以Pairwise Accuracy替代相关性指标，考察文本重叠与GPT-4评判等自动评估方法，揭示其可靠性高度依赖任务、语言及提示中是否包含参考答案。
摘要：
关于指令微调大语言模型（LLM）的研究已采用基于文本重叠和LLM判断的自动方法，作为人类评估的高性价比替代方案。本文对此类方法进行元评估，并评估其在广泛任务中的可靠性。在衡量自动方法与人类评估的一致性时，相关系数指标是最常用的方法，尽管它们在处理并列情况和不同量表时存在固有局限。为解决这些不足，我们使用成对准确率（Pairwise Accuracy）作为标准相关性度量的替代方案。我们观察到，尽管自动评估方法在特定条件下能够近似人类评分，但其有效性高度依赖于具体情境。具体而言，简单的ROUGE-L指标在英语短答案任务中与人类评分相关性很好，但在自由形式生成任务和跨语言场景中并不可靠。使用GPT-4作为评判者这一更先进方法的有效性，会在提示中未包含参考答案时显著下降；而与其他指标相比，该方法恰恰是在这一场景中最有潜力提供最大价值。我们的发现增进了对在开发和评估指令微调LLM时应如何应用与解释自动方法的理解。
英文摘要：
Work on instruction-tuned Large Language Models (LLMs) has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. In this paper, we perform a meta-evaluation of such methods and assess their reliability across a broad range of tasks. In evaluating how well automatic methods align with human evaluations, correlation metrics are the most commonly employed method despite their inherent limitations when dealing with ties and different scales. To address these shortcomings, we use Pairwise Accuracy as an alternative to standard correlation measures. We observe that while automatic evaluation methods can approximate human ratings under specific conditions, their validity is highly context-dependent. Specifically, the simple ROUGE-L metric correlates very well with human ratings for short-answer English tasks but is unreliable in free-form generation tasks and cross-lingual scenarios. The effectiveness of the more advanced method of using GPT-4 as a judge diminishes significantly if reference answers are not included in the prompt, which is the scenario where this method has the potential to provide the most value compared to other metrics. Our findings enhance the understanding of how automatic methods should be applied and interpreted when developing and evaluating instruction-tuned LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
