arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1908.06177cs.LGcs.CLcs.LOstat.ML
收藏

CLUTRR:一个文本归纳推理的诊断基准

CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

McGill University(麦吉尔大学);Montreal Institute of Learning Algorithms (Mila)(蒙特利尔学习算法研究所(Mila));Facebook AI Research (FAIR)(Facebook人工智能研究院(FAIR));Université de Montréal(蒙特利尔大学)

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton

摘要:

自然语言理解(NLU)系统最近的成功一直受到结果的困扰,这些结果强调了这些模型未能以系统和鲁棒的方式进行推广。在这项工作中,我们介绍了一个诊断基准套件,名为CLUTRR,澄清一些关键问题的鲁棒性和系统性的NLU系统。受归纳逻辑程序设计经典著作的启发,CLUTRR要求NLU系统推断短篇小说中人物之间的亲属关系。成功执行此任务需要提取实体之间的关系,以及推断控制这些关系的逻辑规则。CLUTRR允许我们通过评估逻辑规则的组合来精确地测量模型的系统泛化能力,并且它允许我们通过添加策划的噪声事实来评估模型的鲁棒性。我们的实证结果突出了最先进的NLU模型之间的巨大性能差距(例如,BERT和MAC)和一个直接与符号输入一起工作的图神经网络模型--基于图的模型表现出更强的泛化能力和更强的鲁棒性。

英文摘要:

The recent success of natural language understanding (NLU) systems has been troubled by results highlighting the failure of these models to generalize in a systematic and robust way. In this work, we introduce a diagnostic benchmark suite, named CLUTRR, to clarify some key issues related to the robustness and systematicity of NLU systems. Motivated by classic work on inductive logic programming, CLUTRR requires that an NLU system infer kinship relations between characters in short stories. Successful performance on this task requires both extracting relationships between entities, as well as inferring the logical rules governing these relationships. CLUTRR allows us to precisely measure a model's ability for systematic generalization by evaluating on held-out combinations of logical rules, and it allows us to evaluate a model's robustness by adding curated noise facts. Our empirical results highlight a substantial performance gap between state-of-the-art NLU models (e.g., BERT and MAC) and a graph neural network model that works directly with symbolic inputs---with the graph-based model exhibiting both stronger generalization and greater robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑