This paper presents a novel attention-based algorithm for achieving adaptive computation called DACT, which, unlike existing ones, is end-to-end differentiable. Our method can be used in conjunction with many networks; in particular, we study its application to the widely known MAC architecture, obtaining a significant reduction in the number of recurrent steps needed to achieve similar accuracies, therefore improving its performance to computation ratio. Furthermore, we show that by increasing the maximum number of steps used, we surpass the accuracy of even our best non-adaptive MAC in the CLEVR dataset, demonstrating that our approach is able to control the number of steps without significant loss of performance. Additional advantages provided by our approach include considerably improving interpretability by discarding useless steps and providing more insights into the underlying reasoning process. Finally, we present adaptive computation as an equivalent to an ensemble of models, similar to a mixture of expert formulation. Both the code and the configuration files for our experiments are made available to support further research in this area.
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Integrating logical reasoning within deep learning architectures has been a major goal of modern AI systems. In this paper, we propose a new direction toward this goal by introducing a differentiable (smoothed) maximum satisfiability (MAXSAT) solver that can be integrated into the loop of larger deep learning systems. Our (approximate) solver is based upon a fast coordinate descent approach to solving the semidefinite program (SDP) associated with the MAXSAT problem. We show how to analytically differentiate through the solution to this SDP and efficiently solve the associated backward pass. W…
Contemporary neural networks have achieved a series of developments and successes in many aspects; however, when exposed to data outside the training distribution, they may fail to predict correct answers. In this work, we were concerned about this generalisation issue and thus analysed a broad set of models systematically and robustly over long stories. Related experiments were conducted based on the CLUTRR, which is a diagnostic benchmark suite that can analyse generalisation of natural language understanding (NLU) systems by training over small story graphs and testing on larger ones. In or…
University College London（伦敦大学学院）；Facebook AI Research（Facebook人工智能研究院）
Reasoning with knowledge expressed in natural language and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering. General neural architectures that jointly learn representations and transformations of text are very data-inefficient, and it is hard to analyse their reasoning process. These issues are addressed by end-to-end differentiable reasoning systems such as Neural Theorem Provers (NTPs), although they can only be used with small-scale symbolic KBs. In this paper we first propose Greedy NTPs (GNTPs), …
University of Alberta（阿尔伯塔大学）；Yazd University（亚兹德大学）；Sharif University of Technology（谢里夫理工大学）
A strong visual object tracker nowadays relies on its well-crafted modules, which typically consist of manually-designed network architectures to deliver high-quality tracking results. Not surprisingly, the manual design process becomes a particularly challenging barrier, as it demands sufficient prior experience, enormous effort, intuition, and perhaps some good luck. Meanwhile, neural architecture search has gaining grounds in practical applications as a promising method in tackling the issue of automated search of feasible network structures. In this work, we propose a novel cell-level diff…
University College London（伦敦大学学院）；Google DeepMind（谷歌DeepMind）；University of Oxford（牛津大学）
While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend …
We introduce Differentiable Reasoning (DR), a novel semi-supervised learning technique which uses relational background knowledge to benefit from unlabeled data. We apply it to the Semantic Image Interpretation (SII) task and show that background knowledge provides significant improvement. We find that there is a strong but interesting imbalance between the contributions of updates from Modus Ponens (MP) and its logical equivalent Modus Tollens (MT) to the learning process, suggesting that our approach is very sensitive to a phenomenon called the Raven Paradox. We propose a solution to overcom…
We study the problem of learning probabilistic first-order logical rules for knowledge base reasoning. This learning problem is difficult because it requires learning the parameters in a continuous space as well as the structure in a discrete space. We propose a framework, Neural Logic Programming, that combines the parameter and structure learning of first-order logical rules in an end-to-end differentiable model. This approach is inspired by a recently-developed differentiable logic called TensorLog, where inference tasks can be compiled into sequences of differentiable operations. We design…
University of Southern California（南加州大学）；Google Research（谷歌研究院）
Current commonsense reasoning research focuses on developing models that use commonsense knowledge to answer multiple-choice questions. However, systems designed to answer multiple-choice questions may not be useful in applications that do not provide a small list of candidate answers to choose from. As a step towards making commonsense reasoning research more realistic, we propose to study open-ended commonsense reasoning (OpenCSR) -- the task of answering a commonsense question without any pre-defined choices -- using as a resource only a corpus of commonsense facts written in natural langua…
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
Carnegie Mellon University（卡内基梅隆大学）；University of Pennsylvania（宾夕法尼亚大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Petuum Inc.（Petuum公司）
TU Darmstadt（达姆施塔特工业大学）；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）
Reasoning is an essential part of human intelligence and thus has been a long-standing goal in artificial intelligence research. With the recent success of deep learning, incorporating reasoning with deep learning systems, i.e., neuro-symbolic AI has become a major field of interest. We propose the Neuro-Symbolic Forward Reasoner (NSFR), a new approach for reasoning tasks taking advantage of differentiable forward-chaining using first-order logic. The key idea is to combine differentiable forward-chaining reasoning with object-centric (deep) learning. Differentiable forward-chaining reasoning …
This paper introduces a differentiable semantic reasoner, where rules are presented as a relevant set of graph transformations. These rules can be written manually or inferred by a set of facts and goals presented as a training set. While the internal representation uses embeddings in a latent space, each rule can be expressed as a set of predicates conforming to a subset of Description Logic.
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；HKU（香港大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；MIT BCS（麻省理工学院脑与认知科学系）；CBMM（麻省理工学院大脑、心智与机器研究中心）
In this work, we propose a unified framework, called Visual Reasoning with Differ-entiable Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from t…
Learning efficient and interpretable policies has been a challenging task in reinforcement learning (RL), particularly in the visual RL setting with complex scenes. While neural networks have achieved competitive performance, the resulting policies are often over-parameterized black boxes that are difficult to interpret and deploy efficiently. More recent symbolic RL frameworks have shown that high-level domain-specific programming logic can be designed to handle both policy learning and symbolic planning. However, these approaches rely on coded primitives with little feature learning, and whe…
We present efficient differentiable implementations of second-order multi-hop reasoning using a large symbolic knowledge base (KB). We introduce a new operation which can be used to compositionally construct second-order multi-hop templates in a neural model, and evaluate a number of alternative implementations, with different time and memory trade offs. These techniques scale to KBs with millions of entities and tens of millions of triples, and lead to simple models with competitive performance on several learning tasks requiring multi-hop reasoning.
Probabilistic logic reasoning is a central component of such cognitive architectures as OpenCog. However, as an integrative architecture, OpenCog facilitates cognitive synergy via hybridization of different inference methods. In this paper, we introduce a differentiable version of Probabilistic Logic networks, which rules operate over tensor truth values in such a way that a chain of reasoning steps constructs a computation graph over tensors that accepts truth values of premises from the knowledge base as input and produces truth values of conclusions as output. This allows for both learning …
Combining machine learning with logic-based expert systems in order to get the best of both worlds are becoming increasingly popular. However, to what extent machine learning can already learn to reason over rule-based knowledge is still an open problem. In this paper, we explore how symbolic logic, defined as logic programs at a character level, is learned to be represented in a high-dimensional vector space using RNN-based iterative neural networks to perform reasoning. We create a new dataset that defines 12 classes of logic programs exemplifying increased level of complexity of logical rea…
Massachusetts Institute of Technology（麻省理工学院）；Facebook AI Research（Facebook人工智能研究院）
Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills. However, reasoning over mechanics and dexterous interactions can make it challenging to scale learning contact-rich manipulation. In this work, we focus on the problem of visual non-prehensile planar manipulation: given a video of an object in planar motion, find contact-aware robot actions that reproduce the same object motion. We propose a novel architecture, Differentiable Learning for Manipulation (\ours), that combines video decoding neural models with priors from contact mechanics by le…
Simultaneous localization and mapping (SLAM) remains challenging for a number of downstream applications, such as visual robot navigation, because of rapid turns, featureless walls, and poor camera quality. We introduce the Differentiable SLAM Network (SLAM-net) along with a navigation architecture to enable planar robot navigation in previously unseen indoor environments. SLAM-net encodes a particle filter based SLAM algorithm in a differentiable computation graph, and learns task-oriented neural network components by backpropagating through the SLAM algorithm. Because it can optimize all mod…
This paper describes diff-SAT, an Answer Set and SAT solver which combines regular solving with the capability to use probabilistic clauses, facts and rules, and to sample an optimal world-view (multiset of satisfying Boolean variable assignments or answer sets) subject to user-provided probabilistic constraints. The sampling process minimizes a user-defined differentiable objective function using a gradient descent based optimization method called Differentiable Satisfiability Solving ($\partial\mathrm{SAT}$) respectively Differentiable Answer Set Programming ($\partial\mathrm{ASP}$). Use cas…
Queen’s University（女王大学）；Ingenuity Labs Research Institute（Ingenuity Labs研究院）；iFLYTEK Research（科大讯飞研究院）；State Key Laboratory of Cognitive Intelligence（认知智能全国重点实验室）
We explore end-to-end trained differentiable models that integrate natural logic with neural networks, aiming to keep the backbone of natural language reasoning based on the natural logic formalism while introducing subsymbolic vector representations and neural components. The proposed model adapts module networks to model natural logic operations, which is enhanced with a memory component to model contextual information. Experiments show that the proposed framework can effectively model monotonicity-based reasoning, compared to the baseline neural network models without built-in inductive bia…
NeuroSymActive: Differentiable Neural-Symbolic Reasoning with Active Exploration for Knowledge Graph Question Answering
University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）；The Australian National University（澳大利亚国立大学）；Juntendo University（静冈大学）；Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Central South University（中南大学）；Southeast University（东南大学）；China Agricultural University（中国农业大学）；Adelaide University（阿德莱德大学）
TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；Eindhoven University of Technology（埃因霍温理工大学）；German Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
UCL Centre for Artificial Intelligence, University College London（伦敦大学学院人工智能中心）；Facebook AI Research（Facebook 人工智能研究院）
Attempts to render deep learning models interpretable, data-efficient, and robust have seen some success through hybridisation with rule-based systems, for example, in Neural Theorem Provers (NTPs). These neuro-symbolic models can induce interpretable rules and learn representations from data via back-propagation, while providing logical explanations for their predictions. However, they are restricted by their computational complexity, as they need to consider all possible proof paths for explaining a goal, thus rendering them unfit for large-scale applications. We present Conditional Theorem …
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Kuaishou Technology（快手科技）；Shenzhen Institute of Advanced Integration Technology, Shenzhen（深圳先进集成技术研究院）
Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Tencent（腾讯）；Center for Language AI Research, Tohoku University（东北大学语言人工智能研究中心）；RIKEN Center for Advanced Intelligence Project（日本理化学研究所先进人工智能项目中心）
Carney Institute for Brain Science（卡尼脑科学研究所）；Brown University（布朗大学）；Centre de Recherche Cerveau et Cognition（脑与认知研究中心）；CNRS（法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；Artificial and Natural Intelligence Toulouse Institute（图卢兹人工智能与自然智能研究所）
Humans continue to outperform modern AI systems in their ability to flexibly parse and understand complex visual scenes. Here, we present a novel module for visual reasoning, the Guided Attention Model for (visual) Reasoning (GAMR), which instantiates an active vision theory -- positing that the brain solves complex visual reasoning problems dynamically -- via sequences of attention shifts to select and route task-relevant visual information into memory. Experiments on an array of visual reasoning tasks and datasets demonstrate GAMR's ability to learn visual routines in a robust and sample-eff…
Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
ASUS Intelligent Cloud Services (AICS)（华硕智能云服务（AICS））；ASUS Global Pte. Ltd.（华硕全球私人有限公司）；Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心（CFAR））；Institute of High Performance Computing (IHPC)（高性能计算研究所（IHPC））；Agency for Science, Technology and Research (A*STAR)（新加坡科学、技术与研究局（A*STAR））
Microsoft Applied Sciences Group (ASG)（微软应用科学集团）；Microsoft Research AI（微软人工智能研究院）
Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framewo…
Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
Fudan University（复旦大学）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）；Future Living Lab of Alibaba（阿里巴巴未来生活实验室）；University of Southern California（南加州大学）；Shanghai Innovation Institute（上海创新研究院）
Cornell University（康奈尔大学）；Allen Institute for AI（艾伦人工智能研究所）；New York University（纽约大学）；Princeton University（普林斯顿大学）
We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limite…
In this paper we present an approach and a benchmark for visual reasoning in robotics applications, in particular small object grasping and manipulation. The approach and benchmark are focused on inferring object properties from visual and text data. It concerns small household objects with their properties, functionality, natural language descriptions as well as question-answer pairs for visual reasoning queries along with their corresponding scene semantic representations. We also present a method for generating synthetic data which allows to extend the benchmark to other objects or scenes a…
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
School of Artificial Intelligence UCAS（中国科学院大学人工智能学院）；Institute of Automation CAS（中国科学院自动化研究所）；Ant Digital Technologies Ant Group（蚂蚁数字科技蚂蚁集团）；RUC（中国人民大学）；BIT（北京理工大学）
UCLA（加州大学洛杉矶分校）；Stanford University（斯坦福大学）；MIT BCS, CBMM, CSAIL（麻省理工学院脑与认知科学系、心智/大脑/机器研究中心、计算机科学与人工智能实验室）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies. Such composite structures could induce a rich set of semantic concepts and relations, thus playing an important role in the interpretation and organization of visual signals as well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on objects rather than parts. Visual reasoning based on the full part-whole hierarchy is much more challenging than object-cent…
