arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.01940cs.CL
收藏

举办AI会议的最佳地点是哪里?温哥华还是__:为什么完成比较问题很困难

What's the best place for an AI conference, Vancouver or ______: Why completing comparative questions is difficult

Google Research(谷歌研究院)

Avishai Zagoury, Einat Minkov, Idan Szpektor, William W. Cohen

摘要:

虽然像BERT这样的大型神经语言模型(LM)可以被微调,在许多NLP任务上产生最先进的结果,但通常不清楚这些模型实际上学习了什么。在这里,我们研究使用这样的LM来填充人类编写的比较问题中的实体,比如“哪个国家更老,印度还是_?”也就是说,我们研究神经LM提出(而不是回答)合理问题的能力。我们发现,这种填空任务的准确性与人类对问题是否合理的判断密切相关,并且这些模型可以经过训练,在完成三个不同子域的比较问题时达到接近人类水平的性能。然而,分析表明,他们所学到的东西未能对任何关于哪些实体在语义上是可比或相似的广泛概念进行建模-相反,训练的模型是非常特定于领域的,并且性能与训练集中观察到的特定实体之间的共现高度相关。这对于在一般文本语料库上预训练的模型以及在大型比较问题语料库上训练的模型都是如此。因此,我们的研究加强了最近的结果,即很难根据特定基准问题的表现来声称深度模型的世界知识或语言能力。我们公开了我们的评估数据集,以促进未来在人类交互标准下对此类模型中复杂理解和推理的研究。

英文摘要:

Although large neural language models (LMs) like BERT can be finetuned to yield state-of-the-art results on many NLP tasks, it is often unclear what these models actually learn. Here we study using such LMs to fill in entities in human-authored comparative questions, like ``Which country is older, India or ______?'' -- i.e., we study the ability of neural LMs to ask (not answer) reasonable questions. We show that accuracy in this fill-in-the-blank task is well-correlated with human judgements of whether a question is reasonable, and that these models can be trained to achieve nearly human-level performance in completing comparative questions in three different subdomains. However, analysis shows that what they learn fails to model any sort of broad notion of which entities are semantically comparable or similar -- instead the trained models are very domain-specific, and performance is highly correlated with co-occurrences between specific entities observed in the training set. This is true both for models that are pretrained on general text corpora, as well as models trained on a large corpus of comparison questions. Our study thus reinforces recent results on the difficulty of making claims about a deep model's world knowledge or linguistic competence based on performance on specific benchmark problems. We make our evaluation datasets publicly available to foster future research on complex understanding and reasoning in such models at standards of human interaction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑