Multimodal large language models (MLLMs), initiated with a trained LLM, first align images with text and then fine-tune on multimodal mixed inputs. However, the MLLM catastrophically forgets the text-only instructions, which do not include images and can be addressed within the initial LLM. In this paper, we present Wings, a novel MLLM that excels in both text-only dialogues and multimodal comprehension. Analyzing MLLM attention in multimodal instructions reveals that text-only forgetting is related to the attention shifts from pre-image to post-image text. From that, we construct extra modules that act as the boosted learner to compensate for the attention shift. The complementary visual and textual learners, like "wings" on either side, are connected in parallel within each layer's attention block. Initially, image and text inputs are aligned with visual learners operating alongside the main attention, balancing focus on visual elements. Textual learners are later collaboratively integrated with attention-based routing to blend the outputs of the visual and textual learners. We design the Low-Rank Residual Attention (LoRRA) to guarantee high efficiency for learners. Our experimental results demonstrate that Wings outperforms equally-scaled MLLMs in both text-only and visual question-answering tasks. On a newly constructed Interleaved Image-Text (IIT) benchmark, Wings exhibits superior performance from text-only-rich to multimodal-rich question-answering tasks.
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；University of Amsterdam（阿姆斯特丹大学）
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；BIAI, ZJUT & D5Data.ai（BIAI、浙江工业大学及D5Data.ai）；National University of Singapore（新加坡国立大学）
Towards Reasoning-Preserving Unlearning in Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（Mohamed bin Zayed人工智能大学）；University of Queensland（昆士兰大学）；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明责任人工智能与数据分析实验室）；King Abdullah University of Science and Technology (KAUST)（卡塔尔科学与技术大学）
MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University（香港理工大学）；Huawei（华为）
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）
Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Tencent（腾讯）；Center for Language AI Research, Tohoku University（东北大学语言人工智能研究中心）；RIKEN Center for Advanced Intelligence Project（日本理化学研究所先进人工智能项目中心）
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Sun Yat-sen University（中山大学）；Nanyang Technological University（南洋理工大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）
Progressive Multimodal Alignment for Continual Instruction Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；Kyoto University（京都大学）；Migu Culture Technology Co.,Ltd.（咪咕文化科技有限公司）；State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术国家重点实验室）
MAny: Merge Anything for Multimodal Continual Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology（国防科技大学并行与分布式计算国家重点实验室）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；School of Computer Science, Tsinghua University（清华大学计算机科学与技术系）
Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation
Southern University of Science and Technology（南方科技大学）；Microsoft Research Asia（微软亚洲研究院）；Shanghai University of Finance and Economics（上海财经大学）；Peking University（北京大学）
A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Kyoto University（京都大学）；Tencent AI Lab（腾讯AI实验室）；Tomorrow Advancing Life（明天生命科技）
Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU
Aalborg University（奥尔堡大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Milestone Systems（里程碑系统公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）
MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Southeast University（东南大学）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新科学研究院人工智能与机器人中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology（福建 pattern recognition and image understanding 工程学院，厦门大学科技学院）
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia（英属哥伦比亚大学）；Fudan University（复旦大学）；Royal College of Science, Imperial College London（伦敦帝国理工学院皇家科学学院）；Dyson School of Design Engineering（戴森设计工程学院）；Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences（中国科学院苏州生物医学工程技术研究所）；East China Normal University（华东师范大学）
Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Binjiang Research Institute of Zhejiang University（浙江大学滨江研究院）；Linker Technology Research Co. Ltd（领科技术研究有限公司）
