arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.08129cs.CL
收藏

摘要评价是否可以翻译成其他语言?

Does Summary Evaluation Survive Translation to Other Languages?

Primer Technologies Inc.(Primer Technologies公司);Technische Universität Berlin(柏林工业大学)

Spencer Braun, Oleg Vasilyev, Neslihan Iskender, John Bohannon

摘要:

创建高质量的摘要数据集是一项昂贵且耗时的工作,需要经过培训的人员和机器对摘要进行生产和评估。如果在一种语言中做出这样的努力,那么能够在其他语言中使用它而不重复人工注释将是有益的。为了研究我们可以在多大程度上信任这样一个数据集的机器翻译,我们将英语数据集SummEval翻译为七种语言,并比较自动评估措施的性能。我们探讨等效性检验作为适当的统计范式,用于评估人类和自动评分的摘要之间的相关性。虽然我们发现类似于源语言的数据集重用有一定的潜力,但大多数摘要评估方法在翻译过程中并不具有统计等效性。

英文摘要:

The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑