arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.07229cs.AI
收藏

信任RoBERTa胜过BERT:来自CheckListing(自然语言推理任务)的见解

Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task

IIT Bombay(印度理工学院孟买分校);Samsung Korea(三星韩国);Microsoft Research India(微软印度研究院)

Ishan Tarunesh, Somak Aditya, Monojit Choudhury

摘要:

最近的最先进的自然语言理解(NLU)系统往往表现出不可预测的,失败的简单推理的例子。尽管如此,人们对量化具有更可预测行为的系统的进展的关注有限。我们认为推理能力明智的行为总结是弥合这一差距的一步。我们为自然语言推理(NLI)任务(一个代表性的NLU任务)创建了一个CheckList测试套件(184K示例)。我们基准国家的最先进的NLI系统在这个测试套件,它揭示了细粒度的见解推理能力的BERT和ROBERTa。我们的分析进一步揭示了模型在来自相同模板或不同模板但属于相同推理能力的示例上的不一致性,这表明通过对CheckList进行观察来概括模型的行为是不平凡的。通过用户研究,我们发现,用户能够利用行为信息,更好地概括的例子预测从RobERTa,相比,BERT。

英文摘要:

The recent state-of-the-art natural language understanding (NLU) systems often behave unpredictably, failing on simpler reasoning examples. Despite this, there has been limited focus on quantifying progress towards systems with more predictable behavior. We think that reasoning capability-wise behavioral summary is a step towards bridging this gap. We create a CheckList test-suite (184K examples) for the Natural Language Inference (NLI) task, a representative NLU task. We benchmark state-of-the-art NLI systems on this test-suite, which reveals fine-grained insights into the reasoning abilities of BERT and RoBERTa. Our analysis further reveals inconsistencies of the models on examples derived from the same template or distinct templates but pertaining to same reasoning capability, indicating that generalizing the models' behavior through observations made on a CheckList is non-trivial. Through an user-study, we find that users were able to utilize behavioral information to generalize much better for examples predicted from RoBERTa, compared to that of BERT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑