定义与检测人类评估指南中的脆弱性：迈向可靠 NLG 评估的初步研究
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation
AI总结：
本文调查 NLG 人类评估指南的缺失与脆弱性，构建首个标注数据集，提出八类脆弱性分类、编写原则及基于 LLM 的检测方法。
摘要：
人类评估是衡量自然语言生成（NLG）系统质量的黄金标准。然而，评估指南作为确保人类评估可靠且可复现的关键要素，所受关注仍然有限。我们的调查显示，在顶级会议近期涉及人类评估的论文中，仅有 29.84% 公开了其评估指南，而在这些指南中，有 77.09% 被识别出脆弱性。不可靠的评估指南会导致不准确的评估结果，并可能阻碍 NLG 沿着正确方向发展。为应对这些挑战，我们迈出了构建可靠评估指南的第一步，提出了首个人类评估指南数据集：我们收集了对从现有论文中提取的指南以及通过大型语言模型（LLM）生成的指南的标注。随后，我们提出了包含八类脆弱性的分类体系，并制定了编写评估指南的原则。此外，我们还探索了一种利用 LLM 检测指南脆弱性的方法，并提供了一组用于提高人类评估可靠性的建议。带标注的人类评估指南数据集以及脆弱性检测方法的代码均已在线公开。
英文摘要：
Human evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems. Nevertheless, the evaluation guideline, as a pivotal element ensuring reliable and reproducible human assessment, has received limited attention.Our investigation revealed that only 29.84% of recent papers involving human evaluation at top conferences release their evaluation guidelines, with vulnerabilities identified in 77.09% of these guidelines. Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. To address these challenges, we take an initial step towards reliable evaluation guidelines and propose the first human evaluation guideline dataset by collecting annotations of guidelines extracted from existing papers as well as generated via Large Language Models (LLMs). We then introduce a taxonomy of eight vulnerabilities and formulate a principle for composing evaluation guidelines. Furthermore, a method for detecting guideline vulnerabilities has been explored using LLMs, and we offer a set of recommendations to enhance reliability in human evaluation. The annotated human evaluation guideline dataset and code for the vulnerability detection method are publicly available online.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
