SESCORE2:通过合成真实错误学习文本生成评估
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes
AI总结:
本文提出自监督评估指标SESCORE2,通过扰动语料库句子合成真实模型错误进行训练,无需人工评分即可在多种语言和文本生成任务上超越无监督及部分有监督指标。
摘要:
是否可以在没有人工标注评分的情况下训练一个用于评估文本生成质量的通用指标?现有的学习型指标要么在跨文本生成任务中表现不佳,要么需要针对特定任务的人工评分进行训练。在本文中,我们提出了SESCORE2,这是一种用于训练基于模型的文本生成评估指标的自监督方法。其核心思想是通过扰动从语料库中检索到的句子来合成真实的模型错误。SESCORE2的主要优势在于其易于扩展到许多其他语言,同时提供可靠的严重程度估计。我们在三种语言上的四个文本生成任务中评估了SESCORE2和先前的方法。SESCORE2在四个文本生成评估基准上优于无监督指标PRISM,Kendall相关系数提高了0.078。令人惊讶的是,SESCORE2在多个文本生成任务上甚至优于有监督的BLEURT和COMET。代码和数据可在https://github.com/xu1998hz/SEScore2获取。
英文摘要:
Is it possible to train a general metric for evaluating text generation quality without human annotated ratings? Existing learned metrics either perform unsatisfactorily across text generation tasks or require human ratings for training on specific tasks. In this paper, we propose SESCORE2, a self-supervised approach for training a model-based metric for text generation evaluation. The key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus. The primary advantage of the SESCORE2 is its ease of extension to many other languages while providing reliable severity estimation. We evaluate SESCORE2 and previous methods on four text generation tasks across three languages. SESCORE2 outperforms unsupervised metric PRISM on four text generation evaluation benchmarks, with a Kendall improvement of 0.078. Surprisingly, SESCORE2 even outperforms the supervised BLEURT and COMET on multiple text generation tasks. The code and data are available at https://github.com/xu1998hz/SEScore2.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。