如何为NLG模型的高效人工评估选择数据点？
How to Select Datapoints for Efficient Human Evaluation of NLG Models?
AI总结：
本文提出了一套考虑评估成本的数据点选择器，通过指标方差、输出多样性和IRT等方法高效选取人工评估数据，仅需约70%数据即可达到全量数据的评估效果。
摘要：
人工评估是评估文本生成模型的黄金标准。然而，其成本高昂。为了符合预算限制，实践中通常会随机选择测试数据的子集进行人工评估。但是，随机选择的数据可能无法准确代表测试性能，使得该方法在模型比较上缺乏经济效率。因此，在本研究中，我们开发并分析了一套选择器，以获取最具信息量的人工评估数据点，同时将评估成本纳入考量。我们表明，基于自动指标分数方差、模型输出多样性或Item Response Theory的选择器优于随机选择。我们进一步开发了一种方法，将这些选择器提炼到模型输出尚不可用的场景中。具体而言，我们引入了基于源文本的估计器，该估计器仅根据源文本预测项目对人工评估的有用性。我们在两个常见的NLG任务（机器翻译和摘要）中展示了选择器的有效性，并表明仅需约70%的测试数据即可产生与完整数据相同的评估结果。
英文摘要：
Human evaluation is the gold standard for evaluating text generation models. However, it is expensive. In order to fit budgetary constraints, a random subset of the test data is often chosen in practice for human evaluation. However, randomly selected data may not accurately represent test performance, making this approach economically inefficient for model comparison. Thus, in this work, we develop and analyze a suite of selectors to get the most informative datapoints for human evaluation, taking the evaluation costs into account. We show that selectors based on variance in automated metric scores, diversity in model outputs, or Item Response Theory outperform random selection. We further develop an approach to distill these selectors to the scenario where the model outputs are not yet available. In particular, we introduce source-based estimators, which predict item usefulness for human evaluation just based on the source texts. We demonstrate the efficacy of our selectors in two common NLG tasks, machine translation and summarization, and show that only $\sim$70\% of the test data is needed to produce the same evaluation result as the entire data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
