Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.
We study the problem of answering questions about images in the harder setting, where the test questions and corresponding images contain novel objects, which were not queried about in the training data. Such setting is inevitable in real world-owing to the heavy tailed distribution of the visual categories, there would be some objects which would not be annotated in the train set. We show that the performance of two popular existing methods drop significantly (up to 28%) when evaluated on novel objects cf. known objects. We propose methods which use large existing external corpora of (i) unla…
Image text carries essential information to understand the scene and perform reasoning. Text-based visual question answering (text VQA) task focuses on visual questions that require reading text in images. Existing text VQA systems generate an answer by selecting from optical character recognition (OCR) texts or a fixed vocabulary. Positional information of text is underused and there is a lack of evidence for the generated answer. As such, this paper proposes a localization-aware answer prediction network (LaAP-Net) to address this challenge. Our LaAP-Net not only generates the answer to the …
EECS, UC Berkeley（加州大学伯克利分校电气工程与计算机科学系）；University of Amsterdam（阿姆斯特丹大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；Facebook AI Research（Facebook人工智能研究院）
Deep models are the defacto standard in visual decision problems due to their impressive performance on a wide array of visual tasks. On the other hand, their opaqueness has led to a surge of interest in explainable systems. In this work, we emphasize the importance of model explanation in various forms such as visual pointing and textual justification. The lack of data with justification annotations is one of the bottlenecks of generating multimodal explanations. Thus, we propose two large-scale datasets with annotations that visually and textually justify a classification decision for variou…
Department of Automation, Tsinghua University, Beijing, China（清华大学自动化系）；Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；Zhongguancun Academy, Beijing, China（中关村学院）；China Agricultural University, Beijing, China（中国农业大学）；Peking University, Beijing, China（北京大学）；Beijing Institute of Technology, Beijing, China（北京理工大学）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；WeChat Vision, Tencent Inc, Beijing, China（微信视觉，腾讯公司）
Georgia Tech（佐治亚理工学院）；Keio University（庆应义塾大学）；University of Toronto（多伦多大学）
As the computational footprint of modern NLP systems grows, it becomes increasingly important to arrive at more efficient models. We show that by employing graph convolutional document representation, we can arrive at a question answering system that performs comparably to, and in some cases exceeds the SOTA solutions, while using less than 5\% of their resources in terms of trainable parameters. As it currently stands, a major issue in applying GCNs to NLP is document representation. In this paper, we show that a GCN enriched document representation greatly improves the results seen in HotPot…
UMass-Amherst（马萨诸塞大学阿默斯特分校）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research, Cambridge（IBM剑桥研究院）
We explore state-of-the-art neural models for question answering on electronic medical records and improve their ability to generalize better on previously unseen (paraphrased) questions at test time. We enable this by learning to predict logical forms as an auxiliary task along with the main task of answer span detection. The predicted logical forms also serve as a rationale for the answer. Further, we also incorporate medical entity information in these models via the ERNIE architecture. We train our models on the large-scale emrQA dataset and observe that our multi-task entity-enriched mode…
National Taiwan University of Science and Technology（台湾科技大学）
In a spoken multiple-choice question answering (SMCQA) task, given a passage, a question, and multiple choices all in the form of speech, the machine needs to pick the correct choice to answer the question. While the audio could contain useful cues for SMCQA, usually only the auto-transcribed text is utilized in system development. Thanks to the large-scaled pre-trained language representation models, such as the bidirectional encoder representations from transformers (BERT), systems with only auto-transcribed text can still achieve a certain level of performance. However, previous studies hav…
StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
Ant International, Ant Group（蚂蚁集团国际部，蚂蚁集团）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院）；Anhui Provincial Key Laboratory of High Performance Computing（安徽省高性能计算重点实验室）
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
School of Information Technology（信息科技学院）；Monash University Malaysia（墨尔本大学马来西亚分校）；Faculty of Innovation Engineering（创新工程学院）；Macau University of Science and Technology（澳门科学技术大学）；Department of Bioelectronics（生物电子系）；Faculty of Biomedical Engineering（生物医学工程学院）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Many services that perform information retrieval for Points of Interest (POI) utilize a Lucene-based setup with spatial filtering. While this type of system is easy to implement it does not make use of semantics but relies on direct word matches between a query and reviews leading to a loss in both precision and recall. To study the challenging task of semantically enriching POIs from unstructured data in order to support open-domain search and question answering (QA), we introduce a new dataset POIReviewQA. It consists of 20k questions (e.g."is this restaurant dog friendly?") for 1022 Yelp bu…
LHNCBC, U.S. National Library of Medicine（美国国家医学图书馆LHNCBC）；University of South Carolina（南卡罗来纳大学）
One of the cardinal tasks in achieving robust medical question answering systems is textual entailment. The existing approaches make use of an ensemble of pre-trained language models or data augmentation, often to clock higher numbers on the validation metrics. However, two major shortcomings impede higher success in identifying entailment: (1) understanding the focus/intent of the question and (2) ability to utilize the real-world background knowledge to capture the context beyond the sentence. In this paper, we present a novel Medical Knowledge-Enriched Textual Entailment framework that allo…
Computer Vision Center, UAB（巴塞罗那自治大学计算机视觉中心）；Amazon Web Services（亚马逊云科技）
We propose a novel multimodal architecture for Scene Text Visual Question Answering (STVQA), named Layout-Aware Transformer (LaTr). The task of STVQA requires models to reason over different modalities. Thus, we first investigate the impact of each modality, and reveal the importance of the language module, especially when enriched with layout information. Accounting for this, we propose a single objective pre-training scheme that requires only text and spatial cues. We show that applying this pre-training scheme on scanned documents has certain advantages over using natural images, despite th…
VLMs Guided Interpretable Decision Making for Autonomous Driving
Department of Computer Science, Tulane University（路易斯安那大学计算机科学系）；Qualcomm（高通公司）；Department of Industrial and Systems Engineering, North Carolina State University（北卡罗来纳州立大学工业与系统工程系）
Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
University of Limerick（利默里克大学）；Data Driven Computer Engineering Research Centre（数据驱动计算机工程研究中心）；Lero, The Irish Software Research Centre（Lero爱尔兰软件研究中心）；Valeo Vision Systems（瓦莱欧视觉系统）
MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting
Department of Biomedical Engineering（生物医学工程系）；Georgia Institute of Technology（佐治亚理工学院）；Department of Machine Learning（机器学习系）；Department of Radiation Oncology（放射肿瘤科）；Emory University School of Medicine（埃默里大学医学院）；University of Southern California（南加州大学）
Commonsense and background knowledge is required for a QA model to answer many nontrivial questions. Different from existing work on knowledge-aware QA, we focus on a more challenging task of leveraging external knowledge to generate answers in natural language for a given question with context. In this paper, we propose a new neural model, Knowledge-Enriched Answer Generator (KEAG), which is able to compose a natural answer by exploiting and aggregating evidence from all four information sources available: question, passage, vocabulary and knowledge. During the process of answer generation, K…
Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
School of Artificial Intelligence, University of Chinese Academy of Sciences, China（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所MAIS部）；Alibaba Cloud Computing, China（阿里巴巴云计算）
SPARQL Generation: an analysis on fine-tuning OpenLLaMA for Question Answering over a Life Science Knowledge Graph
Department of Ecology and Evolution, University of Lausanne, Switzerland（生态与进化系，洛桑大学，瑞士）；SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland（瑞士生物信息学研究所，洛桑，瑞士）
CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures
Université Côte d’Azur（蔚蓝海岸大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；I3S（信息科学与信号处理实验室）；HiTZ Center - Ixa（HiTZ中心-Ixa团队）；University of the Basque Country UPV/EHU（巴斯克大学）
AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
Indian Institute of Technology（印度理工学院）；Symbiosis Law School（共生法学院）；Indian Institute of Science Education and Research（印度科学教育与研究学院）；University of Birmingham（伯明翰大学）
A Benchmark Dataset with Larger Context for Non-Factoid Question Answering over Islamic Text
School of Electrical Engineering and Computer Science (SEECS)（电气工程与计算机科学学院（SEECS））；National University of Sciences and Technology (NUST)（国家科技大学（NUST））；College of Computer and Information Sciences (CCIS)（计算机与信息科学学院（CCIS））；Prince Sultan University（苏丹王子大学）
Automatic definition extraction from texts is an important task that has numerous applications in several natural language processing fields such as summarization, analysis of scientific texts, automatic taxonomy generation, ontology generation, concept identification, and question answering. For definitions that are contained within a single sentence, this problem can be viewed as a binary classification of sentences into definitions and non-definitions. In this paper, we focus on automatic detection of one-sentence definitions in mathematical texts, which are difficult to separate from surro…
Answer selection (AS) is a critical subtask of the open-domain question answering (QA) problem. The present paper proposes a method called RLAS-BIABC for AS, which is established on attention mechanism-based long short-term memory (LSTM) and the bidirectional encoder representations from transformers (BERT) word embedding, enriched by an improved artificial bee colony (ABC) algorithm for pretraining and a reinforcement learning-based algorithm for training backpropagation (BP) algorithm. BERT can be comprised in downstream work and fine-tuned as a united task-specific architecture, and the pre…
