Compositional Natural Language Inference has been explored to assess the true abilities of neural models to perform NLI. Yet, current evaluations assume models to have full access to all primitive inferences in advance, in contrast to humans that continuously acquire inference knowledge. In this paper, we introduce the Continual Compositional Generalization in Inference (C2Gen NLI) challenge, where a model continuously acquires knowledge of constituting primitive inference tasks as a basis for compositional inferences. We explore how continual learning affects compositional generalization in NLI, by designing a continual learning setup for compositional NLI inference tasks. Our experiments demonstrate that models fail to compositionally generalize in a continual scenario. To address this problem, we first benchmark various continual learning algorithms and verify their efficacy. We then further analyze C2Gen, focusing on how to order primitives and compositional inference types and examining correlations between subtasks. Our analyses show that by learning subtasks continuously while observing their dependencies and increasing degrees of difficulty, continual learning can enhance composition generalization ability.
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
University of Washington(华盛顿大学);Facebook AI Research(脸书人工智能研究院)
关于隐含关系的推理(例如,词对之间的释义、常识、百科全书)对于许多跨句推理问题至关重要。本文提出了新的方法来学习和使用嵌入的词对,隐含地表示背景知识,这种关系。我们的成对嵌入被计算为单词表示的组合函数,这是通过最大化两个单词共同出现的上下文的逐点互信息(PMI)来学习的。我们将这些表示添加到现有推理模型的跨句注意力层(例如,QA的Biorean,NLI的ESIM),而不是扩展或替换现有的词嵌入。实验表明,在最近发布的SQuAD 2.0上的增益为2.7%,在MultiNLI上的增益为1.3%。我们的表示还有助于更好地泛化,在对抗SQuAD数据集上获得约6-7%的收益,在Glockner et al.(2018)的对抗蕴涵测试集上获得8.8%的收益。
Rehearsal-Free Modular and Compositional Continual Learning for Language Models
Bosch Center for Artificial Intelligence(博世人工智能中心);LMU Munich(慕尼黑大学);Hochschule der Medien(斯图加特媒体大学);Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用技术大学)