Multimodal Large Language Models (MLLMs) excel in diverse vision tasks, but full-parameter retraining is computationally expensive as real-world knowledge evolves. Existing continual learning methods often suffer from semantic entanglement in parameter spaces across tasks, impeding the continuous deployment of models. This challenge is especially pronounced in Anomaly Detection (AD), which exhibits triple heterogeneity across modalities, domains, and defect scale variability, significantly complicating multi-task knowledge transfer. In this paper, we propose CL-Anomaly, a parameter-efficient fine-tuning framework based on an isolation-sharing collaboration to enable continual learning for anomaly detection with MLLMs. We introduce the task-private expert PrivLoRA, which physically isolates task-specific subspaces in the parameter space to prevent semantic entanglement of anomaly knowledge in diverse scenarios. The Layer-Adaptive Shared Experts maintain cross-task representations within a unified feature space, enabling knowledge sharing between previous and new tasks. Furthermore, we propose a Layer-Adaptive Knowledge Transfer strategy that automatically selects and dynamically updates the layer-wise key shared experts of each task via a momentum-based mechanism, promoting effective knowledge transfer across related anomaly detection tasks. Extensive experiments across three continual learning scenarios for anomaly detection, including class-incremental, cross-domain, and cross-modal, demonstrate that CL-Anomaly outperforms state-of-the-art methods. Code is available at https://github.com/WenDongyp/CL-Anomaly.
Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济发展实验室);Chongqing Research Institute of HIT(重庆哈工大研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
Anomaly detection in surveillance videos has been recently gaining attention. A challenging aspect of high-dimensional applications such as video surveillance is continual learning. While current state-of-the-art deep learning approaches perform well on existing public datasets, they fail to work in a continual learning framework due to computational and storage issues. Furthermore, online decision making is an important but mostly neglected factor in this domain. Motivated by these research gaps, we propose an online anomaly detection method for surveillance videos using transfer learning and…
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China(计算机科学与技术系,BNRist,清华大学,北京,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国)
CL-BioGAN: Biologically-Inspired Cross-Domain Continual Learning for Hyperspectral Anomaly Detection
Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, School of Computer Science and Technology, Xidian University(协同智能系统重点实验室、教育部、计算机科学与技术学院、西安电子科技大学);Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, School of Artificial Intelligence, Xidian University(智能感知与图像理解重点实验室、教育部、人工智能学院、西安电子科技大学);Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University(协同智能系统重点实验室、教育部、西安电子科技大学)
MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新科学研究院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology(福建 pattern recognition and image understanding 工程学院,厦门大学科技学院)
Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(人工智能大模型关键实验室,人工智能研究院,上海交通大学);vivo Mobile Communication Co., Ltd.(vivo移动通信有限公司)
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education(教育部智能仿生无人系统重点实验室);School of Artificial Intelligence, University of Science and Technology Beijing(北京科技大学人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution
The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);Yale University(耶鲁大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts
College of Computer Science, Sichuan University(四川大学计算机学院);School of Artificial Intelligence, Sichuan University(四川大学人工智能学院);Tianfu Jincheng Laboratory(天府锦城实验室);School of Economics and Management, China University of Petroleum (Beijing)(中国石油大学(北京)经济管理学院)
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models
HKUST(GZ) Deep Interdisciplinary Intelligence Lab(香港科技大学(广州)深度跨学科智能实验室);Wuhan University(武汉大学);Shandong University(山东大学);Tongji University(同济大学);Peking University(北京大学);University of Macau(澳门大学);HKUST(GZ) Institute of Deep Perception Technology, JITRI Deep Interdisciplinary Intelligence Lab(香港科技大学(广州)感知技术研究所,JITRI深度跨学科智能实验室)
University of Southern California(南加州大学);USC Information Sciences Institute(南加州大学信息科学研究所)
Current state-of-the-art vision-and-language models are evaluated on tasks either individually or in a multi-task setting, overlooking the challenges of continually learning (CL) tasks as they arrive. Existing CL benchmarks have facilitated research on task adaptation and mitigating "catastrophic forgetting", but are limited to vision-only and language-only tasks. We present CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL setting, and to systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks. CLiMB incl…
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of Cambridge(剑桥大学);LAION;Jülich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Jülich (FZJ)(于利希研究中心);Helmholtz Association(亥姆霍兹联合会);Google DeepMind(谷歌DeepMind);Munich Center for ML(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学)
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,HKISI-CAS);Huawei Inc(华为公司)
MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Models
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院教育部人工智能重点实验室)
Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Era
Shenzhen Graduate School of Peking University(北京大学深圳研究生院);Centre for Artificial Intelligence and Robotics, HKISI, CAS(香港科学院人工智能与机器人研究中心);Peng Cheng Laboratory(鹏城实验室)
AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Casivision(中科视语);Weiqiao-UCAS Science and Technology Park(魏桥国科科技园)
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology(大连理工大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学)
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia(英属哥伦比亚大学);Fudan University(复旦大学);Royal College of Science, Imperial College London(伦敦帝国理工学院皇家科学学院);Dyson School of Design Engineering(戴森设计工程学院);Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences(中国科学院苏州生物医学工程技术研究所);East China Normal University(华东师范大学)
GCR: Geometry-Consistent Routing for Task-Agnostic Continual Anomaly Detection
Tsinghua University Shenzhen International Graduate School(清华大学深圳国际研究生院);Ratel Soft;Affiliated Fifth Hospital, Wenzhou Medical University(温州医学院附属第五医院);Chinese Medicine Guangdong Laboratory(广东中医药实验室)
One-for-More: Continual Diffusion Model for Anomaly Detection
East China Normal University(东华师范大学);Shanghai Innovation Institute(上海创新研究院);Xiamen University(厦门大学);Shanghai Jiao Tong University(上海交通大学);Tencent YouTu Lab(腾讯YouTu实验室);Tencent WeChatPay Lab33(腾讯微信支付实验室);CATL(宁德时代);Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室)
本文针对扩散模型在持续异常检测中的保真幻觉与灾难性遗忘问题,提出基于梯度投影的持续扩散模型,并通过迭代奇异值分解降低内存开销,同时引入异常掩码网络增强条件机制,在 MVTec 和 VisA 上取得领先性能。
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室);School of Artificial Intelligence, UCAS(人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心)
ReplayCAD: Generative Diffusion Replay for Continual Anomaly Detection
South China University of Technology(华南理工大学);China United Network Communications Corporation Limited Guangdong Branch(中国联合网络通信集团有限公司广东分公司);Pazhou Laboratory(琶洲实验室);Guangdong University of Technology(广东工业大学)
School of Computer Science & Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Computing and Information Technology, Great Bay University(海湾大学计算机与信息学院)
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学)