Compositional generalization is the capability of a model to understand novel compositions composed of seen concepts. There are multiple levels of novel compositions including phrase-phrase level, phrase-word level, and word-word level. Existing methods achieve promising compositional generalization, but the consistency of compositional generalization across multiple levels of novel compositions remains unexplored. The consistency refers to that a model should generalize to a phrase-phrase level novel composition, and phrase-word/word-word level novel compositions that can be derived from it simultaneously. In this paper, we propose a meta-learning based framework, for achieving consistent compositional generalization across multiple levels. The basic idea is to progressively learn compositions from simple to complex for consistency. Specifically, we divide the original training set into multiple validation sets based on compositional complexity, and introduce multiple meta-weight-nets to generate sample weights for samples in different validation sets. To fit the validation sets in order of increasing compositional complexity, we optimize the parameters of each meta-weight-net independently and sequentially in a multilevel optimization manner. We build a GQA-CCG dataset to quantitatively evaluate the consistency. Experimental results on visual question answering and temporal video grounding, demonstrate the effectiveness of the proposed framework. We release GQA-CCG at https://github.com/NeverMoreLCH/CCG.
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
We formalize human language understanding as a structured prediction task where the output is a partially ordered set (poset). Current encoder-decoder architectures do not take the poset structure of semantics into account properly, thus suffering from poor compositional generalization ability. In this paper, we propose a novel hierarchical poset decoding paradigm for compositional generalization in language. Intuitively: (1) the proposed paradigm enforces partial permutation invariance in semantics, thus avoiding overfitting to bias ordering information; (2) the hierarchical mechanism allows …
Multi-Sourced Compositional Generalization in Visual Question Answering
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学，中国）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, China（广东机器感知与智能计算实验室，深圳MSU-BIT大学，中国）
Towards Compositional Generalization of LLMs via Skill Taxonomy Guided Data Synthesis
State Key Laboratory of Complex & Critical Software Environment, Beihang University（复杂与关键软件环境国家重点实验室，北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Compositional Concept Generalization with Variational Quantum Circuits
School of Computer Science；Engineering University of New South Wales Sydney, Australia；Stanford University California, USA；Computer Science University College London London, UK；School of Eng. Maths. \& Tech University of Bristol Bristol, UK；Inst. Logic Language \& Computation University of Amsterdam Amsterdam, NL
Evaluating Morphological Compositional Generalization in Large Language Models
EPFL（瑞士联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Duke University（杜克大学）；Brandeis University（布兰迪斯大学）；LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Johns Hopkins University（约翰霍普金斯大学）；Università della Svizzera Italiana（瑞士意大利大学）；New York University（纽约大学）
Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different distributions of images. Recently, Nikolaus et al. (2019) introduced a dataset to assess compositional generalization in image captioning, where models are evaluated on their ability to describe images with unseen adjective-noun and noun-verb compositions. In this work, we investigate different methods to improve compositional generalization by planning the syntactic structure of a caption. Our experiments show that jo…
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute（创新实验室研究所）；Queen’s University（女王大学）；Smith School of Business（史密斯商学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
School of Computer Science and Engineering, The Hebrew University of Jerusalem（耶路撒冷希伯来大学计算机科学与工程学院）；Department of Computer Science, Ben Gurion University（本·古里安大学计算机科学系）
Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets (Gardneret al., 2020) quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically generate contrast sets for the visual question answering task. Our method comput…
Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（国家多媒体信息处理重点实验室，计算机学院，北京大学）；OPPO AI Center（OPPO人工智能中心）
University of Tübingen（图宾根大学）；University of Copenhagen（哥本哈根大学）；Stanford University（斯坦福大学）
Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts. We study the problem of compositional generalization, which measures how well a model composes unseen combinations of concepts when describing images. State-of-the-art image captioning models show poor generalization performance on this task. We propose a multi-task model to address the poor performance, that combines caption generation and image--sentence ranking, and uses a decoding mechanism that re-ranks the captions according their sim…
Are Object-Centric Representations Better At Compositional Generalization?
Technical University of Munich（慕尼黑技术大学）；Helmholtz AI, Munich（慕尼黑海德堡人工智能研究所）；KTH Royal Institute of Technology（皇家理工学院）；MPI for Intelligent Systems, Tübingen（图宾根人工智能研究所）；Institute of AI for Health, Computational Health Center, Helmholtz Munich（健康人工智能研究所，计算健康中心，海德堡慕尼黑）
Tel-Aviv University（特拉维夫大学）；Allen Institute for AI（艾伦人工智能研究所）
Answering questions that involve multi-step reasoning requires decomposing them and using the answers of intermediate steps to reach the final answer. However, state-of-the-art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out-of-distribution examples. In this work, we propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. Our model induces latent trees, driven by end-to-end (the answer) supervision only. We show that t…
Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心，教育部）；Tianjin University of Science and Technology（天津科技大学）；Beijing Institute of Technology（北京理工大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Deep learning models struggle with compositional generalization, i.e. the ability to recognize or generate novel combinations of observed elementary concepts. In hopes of enabling compositional generalization, various unsupervised learning algorithms have been proposed with inductive biases that aim to induce compositional structure in learned representations (e.g. disentangled representation and emergent language learning). In this work, we evaluate these unsupervised learning algorithms in terms of how well they enable compositional generalization. Specifically, our evaluation protocol focus…
Veermata Jijabai Technological Institute（维尔马塔·吉贾拜技术学院）；Stanford University（斯坦福大学）；University of Washington（华盛顿大学）
Recent video question answering benchmarks indicate that state-of-the-art models struggle to answer compositional questions. However, it remains unclear which types of compositional reasoning cause models to mispredict. Furthermore, it is difficult to discern whether models arrive at answers using compositional reasoning or by leveraging data biases. In this paper, we develop a question decomposition engine that programmatically deconstructs a compositional question into a directed acyclic graph of sub-questions. The graph is designed such that each parent question is a composition of its chil…
TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learning
The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Baptist University（香港浸会大学）；Shenzhen University（深圳大学）；Jilin University（吉林大学）；Hong Kong University of Science and Technology（香港科技大学）
School of Electrical and Computer Engineering, Cornell University（康奈尔大学电气与计算机工程学院）；Siemens Corporate Technology（西门子公司技术部）；Nancy E. and Peter C. Meinig School of Biomedical Engineering, Cornell University（康奈尔大学南希·E·和彼得·C·迈尼格生物医学工程学院）
Compositionality of semantic concepts in image synthesis and analysis is appealing as it can help in decomposing known and generatively recomposing unknown data. For instance, we may learn concepts of changing illumination, geometry or albedo of a scene, and try to recombine them to generate physically meaningful, but unseen data for training and testing. In practice however we often do not have samples from the joint concept space available: We may have data on illumination change in one data set and on geometric change in another one without complete overlap. We pose the following question: …
Monash University（莫纳什大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Technology Sydney（悉尼科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Lab（鹏城实验室）
We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from \charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (S…
NVIDIA Research（英伟达研究院）；Bar-Ilan University（巴伊兰大学）；Technion - Israel Institute of Technology（以色列理工学院）
People easily recognize new visual categories that are new combinations of known components. This compositional generalization capacity is critical for learning in real-world domains like vision and language because the long tail of new combinations dominates the distribution. Unfortunately, learning systems struggle with compositional generalization because they often build on features that are correlated with class labels even if they are not "essential" for the class. This leads to consistent misclassification of samples from a new distribution, like new combinations of known components. He…
Complex visual scenes that are composed of multiple objects, each with attributes, such as object name, location, pose, color, etc., are challenging to describe in order to train neural networks. Usually,deep learning networks are trained supervised by categorical scene descriptions. The common categorical description of a scene contains the names of individual objects but lacks information about other attributes. Here, we use distributed representations of object attributes and vector operations in a vector symbolic architecture to create a full compositional description of a scene in a high-…
