使用COMET的陷阱与展望
Pitfalls and Outlooks in Using COMET
AI总结：
本文系统分析了COMET评估指标在技术、数据和使用报告三方面的陷阱，指出其分数不可比的问题，并提出解决方案及sacreCOMET工具以促进规范使用。
摘要：
COMET指标凭借其与人类翻译质量判断的高度相关性，在机器翻译领域开辟了一条新道路。它的成功源于它是一个经过微调的预训练多语言模型，专门用于质量评估。然而，它作为一个机器学习模型也带来了一系列可能尚未广为人知的陷阱。我们从三个方面研究这些意外行为：1）技术方面：过时的软件版本和计算精度；2）数据方面：测试时的空内容、语言不匹配和翻译腔，以及训练中的分布偏差和领域偏差；3）使用与报告方面：文献中的多参考支持和模型引用。所有这些问题都意味着COMET分数在不同论文之间甚至不同技术设置之间不具有可比性，我们提出了解决每个问题的观点。此外，我们发布了sacreCOMET软件包，它可以为软件和模型配置生成签名以及适当的引用。这项工作的目标是帮助社区更合理地使用COMET指标。
英文摘要：
The COMET metric has blazed a trail in the machine translation community, given its strong correlation with human judgements of translation quality. Its success stems from being a modified pre-trained multilingual model finetuned for quality assessment. However, it being a machine learning model also gives rise to a new set of pitfalls that may not be widely known. We investigate these unexpected behaviours from three aspects: 1) technical: obsolete software versions and compute precision; 2) data: empty content, language mismatch, and translationese at test time as well as distribution and domain biases in training; 3) usage and reporting: multi-reference support and model referencing in the literature. All of these problems imply that COMET scores are not comparable between papers or even technical setups and we put forward our perspective on fixing each issue. Furthermore, we release the sacreCOMET package that can generate a signature for the software and model configuration as well as an appropriate citation. The goal of this work is to help the community make more sound use of the COMET metric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
