通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2110.01013cs.CVcs.AIcs.CLcs.MM
反事实样本合成与训练用于鲁棒的视觉问答
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
The Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）；Columbia University（哥伦比亚大学）；Nanyang Technological University（南洋理工大学）
Long Chen, Yuhang Zheng, Yulei Niu, Hanwang Zhang, Jun Xiao
备注：
IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI 2023. (Extension of CVPR'20 work). arXiv admin note: text overlap with arXiv:2003.06576
Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to complex model design, these ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR, we achieve record-breaking performance on all OOD benchmarks.
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Hikvision Research Institute（海康威视研究院）
Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ide…
The Hebrew University of Jerusalem（希伯来大学）；Google Research（谷歌研究院）；Tel Aviv University（特拉维夫大学）
Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledge - the factual knowledge encoded in the model weights, and (2) contextual knowledge - external knowledge (e.g., a Wikipedia passage) given to the model to generate a grounded answer. Having these two sources of knowledge entangled together is a core issue for generative QA models as it is unclear whether the answer stems from the given non-parametric knowledge or not. This unclarity has implications on issues of trust, interpretability and factuality. In this work, we pro…
A$^2$Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering
Macau University of Science and Technology（澳门科技大学）；Wuhan University of Technology（武汉理工大学）；Nanjing Institute of Technology（南京工程学院）；National Tsing Hua University（国立清华大学）
HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways
University of Calabria（卡拉布里亚大学）；University of Catanzaro（卡坦扎罗大学）；University of Eastern Piedmont（东皮埃蒙特大学）；University of Pisa（比萨大学）；University of Cambridge（剑桥大学）
On Synthesizing Data for Context Attribution in Question Answering
NEC Laboratories Europe（NEC欧洲实验室）；NEC Laboratories America（NEC美国实验室）；NEC Corporation（NEC公司）；University College London（伦敦大学学院）；KU Leuven（鲁汶大学）；Epimind；Center for Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；CAIR, Ss. Cyril and Methodius University of Skopje（斯科普里圣西里尔和美多迪乌斯大学CAIR）
Columbia University（哥伦比亚大学）；HKUST（香港科技大学）；Microsoft Research（微软研究院）；University of California, Los Angeles（加州大学洛杉矶分校）
Vision-language (VL) understanding tasks evaluate models' comprehension of complex visual scenes through multiple-choice questions. However, we have identified two dataset biases that models can exploit as shortcuts to resolve various VL tasks correctly without proper understanding. The first type of dataset bias is \emph{Unbalanced Matching} bias, where the correct answer overlaps the question and image more than the incorrect answers. The second type of dataset bias is \emph{Distractor Similarity} bias, where incorrect answers are overly dissimilar to the correct answer but significantly sim…
While Visual Question Answering (VQA) models continue to push the state-of-the-art forward, they largely remain black-boxes - failing to provide insight into how or why an answer is generated. In this ongoing work, we propose addressing this shortcoming by learning to generate counterfactual images for a VQA model - i.e. given a question-image pair, we wish to generate a new image such that i) the VQA model outputs a different answer, ii) the new image is minimally different from the original, and iii) the new image is realistic. Our hope is that providing such counterfactual examples allows u…
Google Research（谷歌研究院）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Deep NLP models have been shown to learn spurious correlations, leaving them brittle to input perturbations. Recent work has shown that counterfactual or contrastive data -- i.e. minimally perturbed inputs -- can reveal these weaknesses, and that data augmentation using counterfactuals can help ameliorate them. Proposed techniques for generating counterfactuals rely on human annotations, perturbations based on simple heuristics, and meaning representation frameworks. We focus on the task of creating counterfactuals for question answering, which presents unique challenges related to world knowl…
UC Santa Cruz（加州大学圣克鲁兹分校）；UC Santa Barbara（加州大学圣巴巴拉分校）；Google（谷歌）
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel \underline{\textbf{C}}ounterfactual \underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for vision and language models, which simultaneously employs counterfactual generation an…
Question answering (QA) models are well-known to exploit data bias, e.g., the language prior in visual QA and the position bias in reading comprehension. Recent debiasing methods achieve good out-of-distribution (OOD) generalizability with a considerable sacrifice of the in-distribution (ID) performance. Therefore, they are only applicable in domains where the test distribution is known in advance. In this paper, we present a novel debiasing method called Introspective Distillation (IntroD) to make the best of both worlds for QA. Our key technical contribution is to blend the inductive bias of…
CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale
The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室）；Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国）；Nanjing University of Science and Technology（南京理工大学）
University of California, San Diego（加利福尼亚大学圣迭戈分校）；SRI International（SRI国际公司）
In the domain of Visual Question Answering (VQA), studies have shown improvement in users' mental model of the VQA system when they are exposed to examples of how these systems answer certain Image-Question (IQ) pairs. In this work, we show that showing controlled counterfactual image-question examples are more effective at improving the mental model of users as compared to simply showing random examples. We compare a generative approach and a retrieval-based approach to show counterfactual examples. We use recent advances in generative adversarial networks (GANs) to generate counterfactual im…
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Scenario-based question answering (SQA) has attracted an increasing research interest. Compared with the well-studied machine reading comprehension (MRC), SQA is a more challenging task: a scenario may contain not only a textual passage to read but also structured data like tables, i.e., tabular scenario based question answering (TSQA). AI applications of TSQA such as answering multiple-choice questions in high-school exams require synthesizing data in multiple cells and combining tables with texts and domain knowledge to infer answers. To support the study of this task, we construct GeoTSQA. …
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
One of the primary challenges limiting the applicability of deep learning is its susceptibility to learning spurious correlations rather than the underlying mechanisms of the task of interest. The resulting failure to generalise cannot be addressed by simply using more data from the same distribution. We propose an auxiliary training objective that improves the generalization capabilities of neural networks by leveraging an overlooked supervisory signal found in existing datasets. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, …
FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering
King’s College London（伦敦国王学院）；University of Electronic Science and Technology of China（电子科技大学）；Sun Yat-sen University（中山大学）；York University（约克大学）；Southern University of Science and Technology（南方科技大学）；University College London（伦敦大学学院）；Tsinghua University（清华大学）
Facebook AI Research（脸书人工智能研究院）；Georgia Tech（佐治亚理工学院）
We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our resu…
IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；University of South Carolina（南卡罗来纳大学）；IBM Yorktown（IBM约克镇分公司）；Indian AI Research Organization（印度人工智能研究组织）
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
Department of Artificial Intelligence, Chung-Ang University（Chung-Ang大学人工智能系）；Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University（Chung-Ang大学高级成像科学研究院，多媒体与电影）
In order to facilitate natural language understanding, the key is to engage commonsense or background knowledge. However, how to engage commonsense effectively in question answering systems is still under exploration in both research academia and industry. In this paper, we propose a novel question-answering method by integrating multiple knowledge sources, i.e. ConceptNet, Wikipedia, and the Cambridge Dictionary, to boost the performance. More concretely, we first introduce a novel graph-based iterative knowledge retrieval module, which iteratively retrieves concepts and entities related to t…
Scene Graph-Guided Generative AI Framework for Synthesizing and Evaluating Industrial Hazard Scenarios
Ph.D. Student, Dept. of Civil Eng., University of Texas at Arlington. E-mail；Assistant Professor, Dept. of Computer Sci. \& Eng., University of Texas at Arlington. E-mail；Assistant Professor, Dept. of Civil Eng., University of Texas at Arlington. E-mail
DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
Department of Automation, Tsinghua University, Beijing, China（清华大学自动化系）；Zhongguancun Academy, Beijing, China（中关村学院）；School of Software, Xinjiang University, Urumqi, China（新疆大学软件学院）；College of Materials Science and Engineering, Fuzhou University, Fuzhou, China（福州大学材料科学与工程学院）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；Beijing Qianjue Technology Co., Ltd., Beijing, China（北京千 jue 技术有限公司）
Question: I have five fingers but I am not alive. What am I? Answer: a glove. Answering such a riddle-style question is a challenging cognitive process, in that it requires complex commonsense reasoning abilities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural language understanding (NLU). However, there are currently no dedicated datasets aiming to test these abilities. Herein, we present RiddleSense, a new multiple-choice question answering task, which comes with the first large dataset (5.7k examples) for …
DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
Nanjing University of Posts and Telecommunications（南京邮电大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；MAIS, Institute of Automation, Chinese Academy of Sciences（自动化所MAIS）
Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce CRIPP-VQA, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that involve …
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
University of Science and Technology of China（中国科学技术大学）；Data Darkness Lab, MIRACLE Center, USTC（数据黑暗实验室，MIRACLE中心，中国科学技术大学）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
