Mark-Evaluate:使用总体估计方法评估语言生成
Mark-Evaluate: Assessing Language Generation using Population Estimation Methods
摘要:
我们提出了一个家庭的指标来评估语言生成来自广泛使用的生态学人口估计方法。更具体地说,我们使用标记重捕和最大似然方法,已应用在过去几十年来估计封闭种群在野外的大小。我们提出了三个新的指标:ME $_\text{Petersen}$和ME $_\text{CAPTURE}$,它检索一个单值的评估,和ME $_\text{Schnabel}$返回一个双值的度量,以评估评估集的质量和多样性,分别。在合成实验中,我们的方法家族对质量和多样性的下降很敏感。此外,我们的方法显示出更高的相关性,人类的评价比现有的指标在几个具有挑战性的任务,即无条件的语言生成,机器翻译,和文本摘要。
英文摘要:
We propose a family of metrics to assess language generation derived from population estimation methods widely used in ecology. More specifically, we use mark-recapture and maximum-likelihood methods that have been applied over the past several decades to estimate the size of closed populations in the wild. We propose three novel metrics: ME$_\text{Petersen}$ and ME$_\text{CAPTURE}$, which retrieve a single-valued assessment, and ME$_\text{Schnabel}$ which returns a double-valued metric to assess the evaluation set in terms of quality and diversity, separately. In synthetic experiments, our family of methods is sensitive to drops in quality and diversity. Moreover, our methods show a higher correlation to human evaluation than existing metrics on several challenging tasks, namely unconditional language generation, machine translation, and text summarization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。