arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08192cs.CLcs.LG
收藏

KITMUS 测试:评估自然语言理解系统中来自多个来源的知识整合

The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems

McGill University(麦吉尔大学);Mila Quebec AI Institute(米拉魁北克人工智能研究所);Microsoft Research(微软研究院)

Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung

AI总结:

本文提出 KITMUS 测试套件,通过共指消解子任务评估 NLU 模型在多知识来源下的整合与推理能力,发现模型难以实时推理,且特定训练后仍难以可靠整合仅推理时呈现的知识。

摘要:

许多最先进的自然语言理解(NLU)模型基于预训练神经语言模型。这些模型经常利用来自多个来源的信息进行推理。此类推理中重要的一类是那些既需要背景知识(推测包含在模型的预训练参数中),又需要在推理时提供的实例特定信息的推理。然而,NLU 模型在存在多个知识来源的情况下的整合和推理能力在很大程度上研究不足。在这项工作中,我们提出了一个共指消解子任务的测试套件,这些子任务需要对多个事实进行推理。这些子任务在哪些知识来源包含相关事实方面有所不同。我们还引入了仅在推理时使用虚构知识呈现知识的子任务。我们在我们的数据集上评估了最先进的共指消解模型。我们的结果表明,几个模型难以在预训练时和推理时观察到的知识上进行实时推理。然而,通过特定任务训练,一部分模型展示了从多个来源整合某些知识类型的能力。尽管如此,即使是表现最好的模型似乎也难以可靠地整合仅在推理时呈现的知识。

英文摘要:

Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models. These models often make inferences using information from multiple sources. An important class of such inferences are those that require both background knowledge, presumably contained in a model's pretrained parameters, and instance-specific information that is supplied at inference time. However, the integration and reasoning abilities of NLU models in the presence of multiple knowledge sources have been largely understudied. In this work, we propose a test suite of coreference resolution subtasks that require reasoning over multiple facts. These subtasks differ in terms of which knowledge sources contain the relevant facts. We also introduce subtasks where knowledge is present only at inference time using fictional knowledge. We evaluate state-of-the-art coreference resolution models on our dataset. Our results indicate that several models struggle to reason on-the-fly over knowledge observed both at pretrain time and at inference time. However, with task-specific training, a subset of models demonstrates the ability to integrate certain knowledge types from multiple sources. Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑