Humans can progressively learn visual concepts from easy to hard questions. To mimic this efficient learning ability, we propose a competence-aware curriculum for visual concept learning in a question-answering manner. Specifically, we design a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the learning process with an adaptive curriculum. The mIRT effectively estimates the concept difficulty and the model competence at each learning step from accumulated model responses. The estimated concept difficulty and model competence are further utilized to select the most profitable training samples. Experimental results on CLEVR show that with a competence-aware curriculum, the proposed method achieves state-of-the-art performances with superior data efficiency and convergence speed. Specifically, the proposed model only uses 40% of training data and converges three times faster compared with other state-of-the-art methods.
Answering from Sure to Uncertain: Uncertainty-Aware Curriculum Learning for Video Question Answering
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Computer Science & Software Engineering, University of Western Australia（西澳大学计算机科学与软件工程系）；School of Computing and Communications, Lancaster University（兰卡斯特大学计算与通讯学院）；Department of Data Science & AI, Monash University（莫纳什大学数据科学与人工智能系）
Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）；Remote Sensing Technology Institute（遥感技术研究所）
Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model w…
Conservatoire National des Arts et Métiers（法国国立工艺学院）；XXII Group（XXII集团）
Visual Question Answering (VQA) is a complex task requiring large datasets and expensive training. Neural Module Networks (NMN) first translate the question to a reasoning path, then follow that path to analyze the image and provide an answer. We propose an NMN method that relies on predefined cross-modal embeddings to ``warm start'' learning on the GQA dataset, then focus on Curriculum Learning (CL) as a way to improve training and make a better use of the data. Several difficulty criteria are employed for defining CL methods. We show that by an appropriate selection of the CL method the cost…
Visual Question Answering (VQA) is of tremendous interest to the research community with important applications such as aiding visually impaired users and image-based search. In this work, we explore the use of scene graphs for solving the VQA task. We conduct experiments on the GQA dataset which presents a challenging set of questions requiring counting, compositionality and advanced reasoning capability, and provides scene graphs for a large number of images. We adopt image + question architectures for use with scene graphs, evaluate various scene graph generation techniques for unseen image…
MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
A.I.MATICS Inc（A.I.MATICS公司）；Boston University（波士顿大学）；University of Southern California（南加州大学）；Heuron（Heuron公司）；MODULABS, Open Neural Networks Research Lab（MODULABS，开放神经网络研究实验室）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Samsung Electronics（三星电子）
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answering is expensive and the existing benchmarks are relatively small to provide sufficient supervision. To overcome this challenge, this paper proposes a multi-task learning method which is composed of three main components: (1) multi-modal video question answering network that answe…
GRACE: Grounded Reasoning via Adapter Composition and Evidence-Aware Calibration for Educational Visual Question Answering
Columbia University（哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Georgetown University（乔治城大学）；San Francisco State University（旧金山州立大学）；Loughborough University（拉夫堡大学）；Texas A&M University（德克萨斯农工大学）
Facebook AI Research（Facebook 人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）
We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers …
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；IIIS, Tsinghua University（清华大学交叉信息研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Deepmind（深度思考）；MIT BCS（麻省理工学院脑与认知科学系）；CBMM（脑、心智与机器中心）
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns…
