arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.08672cs.CL
收藏

以语用学为中心的自然语言理解评价框架

A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

KU Leuven(鲁汶大学);IRIT (University of Toulouse)(图卢兹大学信息技术与交互研究所)

Damien Sileo, Tim Van-de-Cruys, Camille Pradel, Philippe Muller

摘要:

自然语言理解的新模型最近取得了前所未有的进展,这使得一些研究人员提出,模型诱导通用的文本表示。然而,目前的基准主要是针对语义现象,我们的情况下,语用学需要采取的中心舞台在评估自然语言的理解。我们介绍了PragmEval,一个新的基准评估自然语言理解,团结11语用为重点的评估数据集英语。PragmEval可以在多任务学习设置中用作补充训练数据,并且是公开可用的,以及用于收集和预处理数据集的代码。使用我们的评估套件,我们表明,自然语言推理,一个广泛使用的预训练任务,不会导致真正的通用表示,这对多任务学习提出了新的挑战。

英文摘要:

New models for natural language understanding have recently made an unparalleled amount of progress, which has led some researchers to suggest that the models induce universal text representations. However, current benchmarks are predominantly targeting semantic phenomena; we make the case that pragmatics needs to take center stage in the evaluation of natural language understanding. We introduce PragmEval, a new benchmark for the evaluation of natural language understanding, that unites 11 pragmatics-focused evaluation datasets for English. PragmEval can be used as supplementary training data in a multi-task learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets. Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which presents a new challenge for multi-task learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑