Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like Checking Factual Correctness of Summaries.
Adobe Research, Big Data Experience Lab（Adobe 研究院大数据体验实验室）；National Center for Text Mining, The University of Manchester（曼彻斯特大学国家文本挖掘中心）；IIT Guwahati（印度理工学院古瓦哈提分校）
Learning algorithms for natural language processing (NLP) tasks traditionally rely on manually defined relevant contextual features. On the other hand, neural network models using an only distributional representation of words have been successfully applied for several NLP tasks. Such models learn features automatically and avoid explicit feature engineering. Across several domains, neural models become a natural choice specifically when limited characteristics of data are known. However, this flexibility comes at the cost of interpretability. In this paper, we define three different methods t…
Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities. First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT's pretraining data with additional data related to the def…
We present a novel neural architecture for the Argument Reasoning Comprehension task of SemEval 2018. It is a simple neural network consisting of three parts, collectively judging whether the logic built on a set of given sentences (a claim, reason, and warrant) is plausible or not. The model utilizes contextualized word vectors pre-trained on large machine translation (MT) datasets as a form of transfer learning, which can help to mitigate the lack of training data. Quantitative analysis shows that simply leveraging LSTMs trained on MT datasets outperforms several baselines and non-transferre…
University of Southern California（南加州大学）；IIT Delhi（印度理工学院德里分校）；Tsinghua University（清华大学）；Adobe Research（Adobe研究院）
Recently, knowledge graph (KG) augmented models have achieved noteworthy success on various commonsense reasoning tasks. However, KG edge (fact) sparsity and noisy edge extraction/generation often hinder models from obtaining useful knowledge to reason over. To address these issues, we propose a new KG-augmented model: Hybrid Graph Network (HGN). Unlike prior methods, HGN learns to jointly contextualize extracted and generated knowledge by reasoning over both within a unified graph structure. Given the task input context and an extracted KG subgraph, HGN is trained to generate embeddings for t…
Sun Yat-sen University（中山大学）；Fudan University（复旦大学）；Microsoft Research（微软研究院）；SINOVATION VENTURES（创新工场）
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. In this paper, we study the challenge of analytical reasoning of text and introduce a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task. Furthermore, to address this reasoning challenge, we design two different baselines: (1) a Transformer-based method which le…
Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.
National Computer System Engineering Research Institute of China（中国国家计算机系统工程研究所）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Emotion Recognition in Conversations (ERC) has gained increasing attention for developing empathetic machines. Recently, many approaches have been devoted to perceiving conversational context by deep learning models. However, these approaches are insufficient in understanding the context due to lacking the ability to extract and integrate emotional clues. In this work, we propose novel Contextual Reasoning Networks (DialogueCRN) to fully understand the conversational context from a cognitive perspective. Inspired by the Cognitive Theory of Emotion, we design multi-turn reasoning modules to ext…
Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection
School of Computer Science, University of Sheffield, UK（谢菲尔德大学计算机科学学院）；Healthy Lifespan Institute, University of Sheffield（谢菲尔德大学健康寿命研究所）；Institute of Data Science and Artificial Intelligence, University of Exeter, UK（埃克塞特大学数据科学与人工智能研究所）
Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization
University of Copenhagen（哥本哈根大学）；Technische Universität Berlin（柏林技术大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD – 柏林学习与数据基础研究院）
University of Southern California（南加州大学）；Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；University of California, San Diego（加州大学圣地亚哥分校）
Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning abou…
T$^2$: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering
The Chinese University of Hong Kong（香港中文大学）；University of International Relations（国际关系大学）；Jarvis Research Center, Tencent YouTu Lab（腾讯YouTu实验室）；Westlake University（西湖大学）
Investigating CoT Monitorability in Large Reasoning Models
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析师实验室）；King Abdullah University of Science and Technology（卡布斯大学）；University of Georgia（佐治亚大学）；University of Macau（澳门大学）
Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning
School of Economics and Management, East China Normal University（华东师范大学经济与管理学部）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we propose a generalisation of Contextual Decomposition (GCD). In particular, this setup enables us to accurately distil which part of a prediction stems from semantic heuristics, which part truly emanates from syntactic cues and which part arise from the model biases themselves instea…
Singapore University of Technology and Design（新加坡科技设计大学）；University of Michigan（密歇根大学）
This paper addresses the problem of dialogue reasoning with contextualized commonsense inference. We curate CICERO, a dataset of dyadic conversations with five types of utterance-level reasoning-based inferences: cause, subsequent event, prerequisite, motivation, and emotional reaction. The dataset contains 53,105 of such inferences from 5,672 dialogues. We use this dataset to solve relevant generative and discriminative tasks: generation of cause and subsequent event; generation of prerequisite, motivation, and listener's emotional reaction; and selection of plausible alternatives. Our result…
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China（中国教育部机器学习与产业智能工程研究中心）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's…
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
NUS Department of Computer Science（国立新加坡大学计算机科学系）；MIT EECS（麻省理工学院电子工程与计算机科学系）；A*STAR（新加坡科技研究局）；Singapore-MIT Alliance for Research and Technology (SMART)（新加坡-麻省理工联合研究技术机构（SMART））
