arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.07290cs.CL
收藏

基于通信的自然语言生成评价

Communication-based Evaluation for Natural Language Generation

Stanford University(斯坦福大学)

Benjamin Newman, Reuben Cohn-Gordon, Christopher Potts

摘要:

自然语言生成(NLG)系统通常使用n-gram重叠度量(例如BLEU、ROUGE)来评估。这些措施并不直接捕捉语义或说话者的意图,因此它们往往与我们对NLG的真正目标不一致。在这项工作中,我们认为,而不是基于通信的评估:假设一个NLG系统的目的是传达信息给读者/听众,我们可以直接评估其有效性在这项任务中使用的理性言语行为模型的语用语言使用。我们用包含预定义质量类别描述的颜色参考数据集进行说明,表明我们的方法比任何突出的n-gram重叠方法更好地与这些质量类别对齐。

英文摘要:

Natural language generation (NLG) systems are commonly evaluated using n-gram overlap measures (e.g. BLEU, ROUGE). These measures do not directly capture semantics or speaker intentions, and so they often turn out to be misaligned with our true goals for NLG. In this work, we argue instead for communication-based evaluations: assuming the purpose of an NLG system is to convey information to a reader/listener, we can directly evaluate its effectiveness at this task using the Rational Speech Acts model of pragmatic language use. We illustrate with a color reference dataset that contains descriptions in pre-defined quality categories, showing that our method better aligns with these quality categories than do any of the prominent n-gram overlap methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑