迈向更好的指令遵循评估：以摘要生成任务为例的案例研究
Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization
AI总结：
针对LLMs指令遵循评估方法正确性研究不足的问题，本文构建了含300个文档-指令对、900个答案的真实数据集riSum进行元评估，并提出新的基于LLM的无参考评估方法，其表现媲美高成本有参考指标。
摘要：
尽管近期取得了进展，但评估大型语言模型（LLMs）遵循用户指令的程度仍是一个开放问题。虽然语言模型的评估方法中基于提示的方法有所增加，但针对这些方法正确性的研究仍然有限。在本研究中，我们对多种指标进行元评估，以量化它们测量LLMs指令遵循能力的准确度。我们的研究基于基于查询的接地摘要任务进行，通过收集一个新的短格式真实世界数据集riSum来展开，该数据集包含300个文档-指令对，每个对应3个答案。所有900个答案均由3名人类标注者进行评分。利用riSum，我们分析了评估方法与人类判断之间的一致性。最后，我们提出了新的基于LLM的无参考评估方法，这些方法改进了已建立的基线，并且与需要高质量摘要的高成本有参考指标表现相当。
英文摘要：
Despite recent advances, evaluating how well large language models (LLMs) follow user instructions remains an open problem. While evaluation methods of language models have seen a rise in prompt-based approaches, limited work on the correctness of these methods has been conducted. In this work, we perform a meta-evaluation of a variety of metrics to quantify how accurately they measure the instruction-following abilities of LLMs. Our investigation is performed on grounded query-based summarization by collecting a new short-form, real-world dataset riSum, containing 300 document-instruction pairs with 3 answers each. All 900 answers are rated by 3 human annotators. Using riSum, we analyze the agreement between evaluation methods and human judgment. Finally, we propose new LLM-based reference-free evaluation methods that improve upon established baselines and perform on par with costly reference-based metrics that require high-quality summaries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
