通过稳定排名概率寻找可复现的人类评估
Finding Replicable Human Evaluations via Stable Ranking Probability
AI总结:
本文以机器翻译的MQM评估框架为案例,研究评分分配、评分数量和分数归一化的最优配置,为设计可复现的人类评估提供具体建议,并发布了包含近14万条标注的最大公开多段翻译评分数据集。
摘要:
可靠的人类评估对于开发成功的自然语言生成模型至关重要,但实现这一目标极其困难。在按质量对系统进行排名时,稳定性是一项关键要求:在重复评估中系统排名保持一致不仅是可取的,而且是必不可少的。没有稳定性,就没有可靠的基础来进行爬山式优化或产品发布决策。在本文中,我们以机器翻译及其最先进的人类评估框架MQM作为案例研究,以理解如何建立可靠的人类评估从而得出稳定的结论。我们研究了项目分配给评分者的最优配置、每个项目的评分数量以及分数归一化。我们在两个语言对上的研究为设计可复现的人类评估研究提供了具体建议。我们还收集并发布了最大的公开可用的多段翻译数据集,该数据集由多位专业译员进行评分,包含两个语言对上近140,000个片段标注。
英文摘要:
Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。