Visual Question Answering (VQA) is a multi-discipline research task. To produce the right answer, it requires an understanding of the visual content of images, the natural language questions, as well as commonsense reasoning over the information contained in the image and world knowledge. Recently, large-scale Vision-and-Language Pre-trained Models (VLPMs) have been the mainstream approach to VQA tasks due to their superior performance. The standard practice is to fine-tune large-scale VLPMs pre-trained on huge general-domain datasets using the domain-specific VQA datasets. However, in reality, the application domain can change over time, necessitating VLPMs to continually learn and adapt to new domains without forgetting previously acquired knowledge. Most existing continual learning (CL) research concentrates on unimodal tasks, whereas a more practical application scenario, i.e, CL on cross-domain VQA, has not been studied. Motivated by this, we introduce CL-CrossVQA, a rigorous Continual Learning benchmark for Cross-domain Visual Question Answering, through which we conduct extensive experiments on 4 VLPMs, 4 CL approaches, and 5 VQA datasets from different domains. In addition, by probing the forgetting phenomenon of the intermediate layers, we provide insights into how model architecture affects CL performance, why CL approaches can help mitigate forgetting in VLPMs to some extent, and how to design CL approaches suitable for VLPMs in this challenging continual learning environment. To facilitate future work on CL for cross-domain VQA, we will release our datasets and code.
Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London(伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心(WEISS));The Chinese University of Hong Kong (CUHK)(香港中文大学(CUHK));Shun Hing Institute of Advanced Engineering, CUHK(香港中文大学信兴高等工程研究所)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Shanghai Open University(上海开放大学);ZhuQingTing Data Technology (Zhejiang) Co., Ltd(朱清庭数据科技(浙江)有限公司)
Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering
LTCI, Télécom-Paris, Institut Polytechnique de Paris(LTCI, Télécom-Paris, Institut Polytechnique de Paris);Inria, LJK, Univ. Grenoble Alpes(Inria, LJK, 火车头 Grenoble Alpes);Universitat Autónoma de Barcelona(Autonomous University of Barcelona)
HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering
School of Computer Science, Hubei University, Wuhan 430062, China(湖北大学计算机学院,武汉430062,中国);Hubei Key Laboratory of Big Data Intelligent Analysis and Application (Hubei University), Wuhan 430062, China(湖北省大数据智能分析与应用重点实验室(湖北大学),武汉430062,中国);Key Laboratory of Intelligent Sensing System and Security (Hubei University), Ministry of Education, Wuhan 430062, China(智能感知系统与安全重点实验室(湖北大学),教育部,武汉430062,中国)