TVQA is a large scale video question answering (video-QA) dataset based on popular TV shows. The questions were specifically designed to require "both vision and language understanding to answer". In this work, we demonstrate an inherent bias in the dataset towards the textual subtitle modality. We infer said bias both directly and indirectly, notably finding that models trained with subtitles learn, on-average, to suppress video feature contribution. Our results demonstrate that models trained on only the visual information can answer ~45% of the questions, while using only the subtitles achieves ~68%. We find that a bilinear pooling based joint representation of modalities damages model performance by 9% implying a reliance on modality specific information. We also show that TVQA fails to benefit from the RUBi modality bias reduction technique popularised in VQA. By simply improving text processing using BERT embeddings with the simple model first proposed for TVQA, we achieve state-of-the-art results (72.13%) compared to the highly complex STAGE model (70.50%). We recommend a multimodal evaluation framework that can highlight biases in models and isolate visual and textual reliant subsets of data. Using this framework we propose subsets of TVQA that respond exclusively to either or both modalities in order to facilitate multimodal modelling as TVQA originally intended.
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Samsung Research（三星研究院）
This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most approp…
To understand movies, humans constantly reason over the dialogues and actions shown in specific scenes and relate them to the overall storyline already seen. Inspired by this behaviour, we design ROLL, a model for knowledge-based video story question answering that leverages three crucial aspects of movie understanding: dialog comprehension, scene reasoning, and storyline recalling. In ROLL, each of these tasks is in charge of extracting rich and diverse information by 1) processing scene dialogues, 2) generating unsupervised video scene descriptions, and 3) obtaining external knowledge in a w…
We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an…
Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；City University of Hong Kong（香港城市大学）；Sun Yat-sen University（中山大学）；Harbin Institute of Technology（哈尔滨工业大学）；Nanyang Technological University（新加坡国立大学）
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Institute of Information Engineering, Chinese Academy of Sciences（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（网络安全学院，中国科学院大学）
Detecting Dataset Bias in Medical AI: A Generalized and Modality-Agnostic Auditing Framework
The Johns Hopkins University（约翰霍普金斯大学）；The Johns Hopkins University Applied Physics Laboratory（约翰霍普金斯大学应用物理实验室）；Center for Devices and Radiological Health, U.S. Food and Drug Administration（美国食品药品监督管理局设备与放射健康中心）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Samsung Electronics（三星电子）
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answering is expensive and the existing benchmarks are relatively small to provide sufficient supervision. To overcome this challenge, this paper proposes a multi-task learning method which is composed of three main components: (1) multi-modal video question answering network that answe…
Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of California, Merced（加州大学梅德福分校）；University of California, San Diego（加州大学圣地亚哥分校）
The task of language-guided video temporal grounding is to localize the particular video clip corresponding to a query sentence in an untrimmed video. Though progress has been made continuously in this field, some issues still need to be resolved. First, most of the existing methods rely on the combination of multiple complicated modules to solve the task. Second, due to the semantic gaps between the two different modalities, aligning the information at different granularities (local and global) between the video and the language is significant, which is less addressed. Last, previous works do…
Vision-Language Pre-training (VLP) models have achieved state-of-the-art performance in numerous cross-modal tasks. Since they are optimized to capture the statistical properties of intra- and inter-modality, there remains risk to learn social biases presented in the data as well. In this work, we (1) introduce a counterfactual-based bias measurement \emph{CounterBias} to quantify the social bias in VLP models by comparing the [MASK]ed prediction probabilities of factual and counterfactual samples; (2) construct a novel VL-Bias dataset including 24K image-text pairs for measuring gender bias i…
Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion
Institute of High Performance Computing, Agency for Science, Technology & Research（新加坡科技研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology & Research（新加坡科技研究局前沿人工智能研究中心）；Institute for Infocomm Research, Agency for Science, Technology & Research（新加坡科技研究局信息通信研究所）
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
China University of Mining and Technology（中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education（矿山数字化教育部工程研究中心）；University of Auckland（奥克兰大学）
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院（KAIST））；Samsung Research（三星研究院）
This paper proposes the progressive attention memory network (PAMN) for movie story question answering (QA). Movie story QA is challenging compared to VQA in two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour, (2) it has both video and subtitle where different questions require different modality to infer the answer. To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant tem…
National University of Singapore（新加坡国立大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Shandong University（山东大学）；Shandong Artificial Intelligence Institute（山东人工智能研究院）；University of Florence（佛罗伦萨大学）
Making each modality in multi-modal data contribute is of vital importance to learning a versatile multi-modal model. Existing methods, however, are often dominated by one or few of modalities during model training, resulting in sub-optimal performance. In this paper, we refer to this problem as modality bias and attempt to study it in the context of multi-modal classification systematically and comprehensively. After stepping into several empirical analysis, we recognize that one modality affects the model prediction more just because this modality has a spurious correlation with instance lab…
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization
UVLab, Department of Computer Science, University of Warwick（华威大学计算机科学系UVLab）；Department of Automation, University of Cambridge（剑桥大学自动化系）；Department of Computer Science, The University of Sheffield（谢菲尔德大学计算机科学系）
Dual-level Modality Debiasing Learning for Unsupervised Visible-Infrared Person Re-Identification
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Video search has become the main routine for users to discover videos relevant to a text query on large short-video sharing platforms. During training a query-video bi-encoder model using online search logs, we identify a modality bias phenomenon that the video encoder almost entirely relies on text matching, neglecting other modalities of the videos such as vision, audio. This modality imbalanceresults from a) modality gap: the relevance between a query and a video text is much easier to learn as the query is also a piece of text, with the same modality as the video text; b) data bias: most t…
Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection
The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
Indian Institute of Technology, Bombay（印度理工学院孟买分校）
We focus on the audio-visual video parsing (AVVP) problem that involves detecting audio and visual event labels with temporal boundaries. The task is especially challenging since it is weakly supervised with only event labels available as a bag of labels for each video. An existing state-of-the-art model for AVVP uses a hybrid attention network (HAN) to generate cross-modal features for both audio and visual modalities, and an attentive pooling module that aggregates predicted audio and visual segment-level event probabilities to yield video-level event probabilities. We provide a detailed ana…
Massachusetts Eye and Ear（麻省眼耳医院）；Harvard Medical School（哈佛医学院）；Florida International University（佛罗里达国际大学）
Neuroimaging is essential in brain studies for the diagnosis and identification of disease, structure, and function of the brain in its healthy and disease states. Literature shows that there are advantages of multitasking with some deep learning (DL) schemes in challenging neuroimaging applications. This study examines the feasibility of using multitasking in three different applications, including tissue segmentation, cross-modality conversion, and bias-field correction. These applications reflect five different scenarios in which multitasking is explored and 280 training and testing session…
ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Group Activity Recognition detects the activity collectively performed by a group of actors, which requires compositional reasoning of actors and objects. We approach the task by modeling the video as tokens that represent the multi-scale semantic concepts in the video. We propose COMPOSER, a Multiscale Transformer based architecture that performs attention-based reasoning over tokens at each scale and learns group activity compositionally. In addition, prior works suffer from scene biases with privacy and ethical concerns. We only use the keypoint modality which reduces scene biases and preve…
Texas A&M University（德州农工大学）；Google Research（谷歌研究院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Celonis Inc.（赛洛尼斯公司）；CeloAI
Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs. Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned. For example, even in the commonly adopted instructional videos, a speaker can sometimes refer to something that is not visually p…
Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（前沿人工智能研究中心，科技研究局，新加坡）；Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；National University of Singapore, Singapore（新加坡国立大学）；Zhejiang University, China（浙江大学）
Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to b…
