arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.03189cs.CLcs.LG
收藏

人或机器:自动评估NLG文本的人的可能性

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts

Charles University in Prague(布拉格查理大学)

Erion Çano, Ondřej Bojar

摘要:

由数据驱动的智能方法产生的各种文本质量标准的自动评估是非常常见和有用的,因为它便宜,快速,并且通常会产生可重复的结果。在本文中,我们提出了一种尝试,以自动化的人类喜欢的评价输出文本样本来自自然语言生成方法,用于解决几个任务。我们建议使用一个人类喜欢度分数,它显示了一个方法的输出样本中看起来像是由人类编写的样本的百分比。我们没有让人类参与者对这些样本进行标记或评级,而是通过使用基于大型预训练语言模型及其概率分布的判别程序来完全自动化该过程。作为后续,我们计划对人类书写和机器生成的文本进行实证分析,以找到这种评估方法的最佳设置。涉及人类参与者的验证过程还将检查自动评估与人类判断的相关性。

英文摘要:

Automatic evaluation of various text quality criteria produced by data-driven intelligent methods is very common and useful because it is cheap, fast, and usually yields repeatable results. In this paper, we present an attempt to automate the human likeliness evaluation of the output text samples coming from natural language generation methods used to solve several tasks. We propose to use a human likeliness score that shows the percentage of the output samples from a method that look as if they were written by a human. Instead of having human participants label or rate those samples, we completely automate the process by using a discrimination procedure based on large pretrained language models and their probability distributions. As follow up, we plan to perform an empirical analysis of human-written and machine-generated texts to find the optimal setup of this evaluation approach. A validation procedure involving human participants will also check how the automatic evaluation correlates with human judgments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑