在WMT 2019上重新评估机器翻译中的人类平等和超人性能
Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019
摘要：
我们重新评估了在WMT 2019新闻共享任务中提出的人类平等和超人表现的说法，用于三个翻译方向：英语到德语，英语到俄语和德语到英语。首先，我们确定了人类评估该共享任务时的三个潜在问题：（i）可用的句间上下文数量有限，（ii）评估者的翻译能力有限，（iii）参考翻译的使用。然后，我们进行修改后的评估，考虑到这些问题。我们的研究结果表明，所有在WMT 2019上提出的人类平等和超人表现的说法都应该被驳斥，除了英语到德语的人类平等的说法。基于我们的研究结果，我们提出了一套建议和开放的问题，为未来的评估人类平等的机器翻译。
英文摘要：
We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human performance made at WMT 2019 should be refuted, except the claim of human parity for English-to-German. Based on our findings, we put forward a set of recommendations and open questions for future assessments of human parity in machine translation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
