穿越指标迷宫:调和分数幅度与准确率
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
AI总结:
本文研究现代机器翻译指标的动态范围,借助新数据集 ToShip23 建立指标分数增量与人类可感知系统差异之间的对应关系,并提出比 p 值更稳定的增量-准确率评估方法。
摘要:
十年前,单一指标 BLEU 主导了机器翻译研究的进展。无论好坏,如今已不存在这样的共识,因此研究人员很难形成并保持那种曾推动早期研究和部署决策的、关于指标增量的启发式直觉。本文研究多种现代指标的“动态范围”,旨在提供对指标内部及指标之间分数差异含义的集体理解;换言之,我们提出如下问题:两个系统之间需要多大的指标 Y 分数差异 X,人类才能察觉?我们在一个新的大型数据集 ToShip23 上进行评估,利用该数据集发现指标在系统层面达到对人类有意义的差异时所对应的增量,并通过成对系统准确率来衡量。我们还进一步表明,就测试集规模而言,这种建立增量-准确率关系的方法比标准的统计 p 值使用方式更稳定。在数据规模允许的情况下,我们还探讨了指标增量与准确率在更细粒度特征上的影响,例如翻译方向、领域和系统接近程度。
英文摘要:
Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for researchers to develop and retain the kinds of heuristic intuitions about metric deltas that drove earlier research and deployment decisions. This paper investigates the "dynamic range" of a number of modern metrics in an effort to provide a collective understanding of the meaning of differences in scores both within and among metrics; in other words, we ask what point difference X in metric Y is required between two systems for humans to notice? We conduct our evaluation on a new large dataset, ToShip23, using it to discover deltas at which metrics achieve system-level differences that are meaningful to humans, which we measure by pairwise system accuracy. We additionally show that this method of establishing delta-accuracy is more stable than the standard use of statistical p-values in regards to testset size. Where data size permits, we also explore the effect of metric deltas and accuracy across finer-grained features such as translation direction, domain, and system closeness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。