Self-supervised learning is an efficient pre-training method for medical image analysis. However, current research is mostly confined to specific-modality data pre-training, consuming considerable time and resources without achieving universality across different modalities. A straightforward solution is combining all modality data for joint self-supervised pre-training, which poses practical challenges. Firstly, our experiments reveal conflicts in representation learning as the number of modalities increases. Secondly, multi-modal data collected in advance cannot cover all real-world scenarios. In this paper, we reconsider versatile self-supervised learning from the perspective of continual learning and propose MedCoSS, a continuous self-supervised learning approach for multi-modal medical data. Unlike joint self-supervised learning, MedCoSS assigns different modality data to different training stages, forming a multi-stage pre-training process. To balance modal conflicts and prevent catastrophic forgetting, we propose a rehearsal-based continual learning method. We introduce the k-means sampling strategy to retain data from previous modalities and rehearse it when learning new modalities. Instead of executing the pretext task on buffer data, a feature distillation strategy and an intra-modal mixup strategy are applied to these data for knowledge retention. We conduct continuous self-supervised pre-training on a large-scale multi-modal unlabeled dataset, including clinical reports, X-rays, CT scans, MRI scans, and pathological images. Experimental results demonstrate MedCoSS's exceptional generalization ability across nine downstream datasets and its significant scalability in integrating new modality data. Code and pre-trained weight are available at https://github.com/yeerwen/MedCoSS.
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Xi’an Jiaotong University（西安交通大学）；School of Information and Communications Engineering, Xi’an Jiaotong University（信息与通信工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院）
Multimodal representations and continual learning are two areas closely related to human intelligence. The former considers the learning of shared representation spaces where information from different modalities can be compared and integrated (we focus on cross-modal retrieval between language and visual representations). The latter studies how to prevent forgetting a previously learned task when learning a new one. While humans excel in these two aspects, deep neural networks are still quite limited. In this paper, we propose a combination of both problems into a continual cross-modal retrie…
CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusion
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Future Science and Engineering, Soochow University（未来科学与工程学院，苏州大学）；DataLab: Data Science and Informatics, University of California, Davis（数据实验室：数据科学与信息学，加州大学戴维斯分校）；University of Rochester（罗切斯特大学）；Ningbo University（宁波大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）；Academy for Computer Science and Informatics, Cardiff University（计算机科学与信息学学院，卡迪夫大学）
Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Jiao Tong University（上海交通大学）；Sichuan University Pittsburgh Institute（四川大学匹兹堡学院）
Inevitable domain and task discrepancies in real-world scenarios can impair the generalization performance of the pre-trained deep models for medical data. Therefore, we audaciously propose that we should build a general-purpose medical AI system that can be seamlessly adapted to downstream domains/tasks. Since the domain/task adaption procedures usually involve additional labeling work for the target data, designing a data-efficient adaption algorithm is desired to save the cost of transferring the learned knowledge. Our recent work found that vision-language models (VLMs) are efficient learn…
Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）
LMU München（慕尼黑大学）；Technische Universität München（慕尼黑工业大学）
The ability of neural networks to continuously learn and adapt to new tasks while retaining prior knowledge is crucial for many applications. However, current neural networks tend to forget previously learned tasks when trained on new ones, i.e., they suffer from Catastrophic Forgetting (CF). The objective of Continual Learning (CL) is to alleviate this problem, which is particularly relevant for medical applications, where it may not be feasible to store and access previously used sensitive patient data. In this work, we propose a Continual Learning approach for brain segmentation, where a si…
Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models
College of Intelligent Robotics and Advanced Manufacturing（智能机器人与先进制造学院）；Fudan University（复旦大学）；Fysics Intelligence Technologies Co., Ltd. (Fysics AI)（Fysics智能技术有限公司（Fysics AI））；School of Computer Science and Engineering（计算机科学与工程学院）；Central South University（中南大学）
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China（计算机科学与技术系，BNRist，清华大学，北京，中国）；Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）
University of Amsterdam（阿姆斯特丹大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
Deep learning models have shown a great effectiveness in recognition of findings in medical images. However, they cannot handle the ever-changing clinical environment, bringing newly annotated medical data from different sources. To exploit the incoming streams of data, these models would benefit largely from sequentially learning from new samples, without forgetting the previously obtained knowledge. In this paper we introduce LifeLonger, a benchmark for continual disease classification on the MedMNIST collection, by applying existing state-of-the-art continual learning methods. In particular…
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
Taming Modality Entanglement in Continual Audio-Visual Segmentation
School of Artificial Intelligence, UCAS（人工智能学院，UCAS）；MAIS, Institute of Automation（自动化研究所MAIS）；School of Intelligent Science and Technology, University of Science and Technolog Beijing（智能科学与技术学院，北京理工大学）
Augmenting Continual Learning of Diseases with LLM-Generated Visual Concepts
Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Peng Cheng Laboratory（鹏城实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, MOE（教育部机器智能与先进计算重点实验室）
SDC-UDA: Volumetric Unsupervised Domain Adaptation Framework for Slice-Direction Continuous Cross-Modality Medical Image Segmentation
Yonsei University（延世大学）；Harvard Medical School（哈佛医学院）；Naver AI Lab（Naver人工智能实验室）；Naver Cloud（Naver云）；Probe Medical Inc.（Probe医疗公司）；Martinos Center for Biomedical Imaging（马丁诺斯生物医学成像中心）；Korea Institute of Science and Technology（韩国科学技术研究院）
Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Yale University（耶鲁大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Deep learning for medical imaging suffers from temporal and privacy-related restrictions on data availability. To still obtain viable models, continual learning aims to train in sequential order, as and when data is available. The main challenge that continual learning methods face is to prevent catastrophic forgetting, i.e., a decrease in performance on the data encountered earlier. This issue makes continuous training of segmentation models for medical applications extremely difficult. Yet, often, data from at least two different domains is available which we can exploit to train the model i…
NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts
College of Computer Science, Sichuan University（四川大学计算机学院）；School of Artificial Intelligence, Sichuan University（四川大学人工智能学院）；Tianfu Jincheng Laboratory（天府锦城实验室）；School of Economics and Management, China University of Petroleum (Beijing)（中国石油大学（北京）经济管理学院）
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Tianjin University（天津大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sheffield（谢菲尔德大学）
Vision Language Models (VLMs), pre-trained on large-scale image-text datasets, enable zero-shot predictions for unseen data but may underperform on specific unseen tasks. Continual learning (CL) can help VLMs effectively adapt to new data distributions without joint training, but faces challenges of catastrophic forgetting and generalization forgetting. Although significant progress has been achieved by distillation-based methods, they exhibit two severe limitations. One is the popularly adopted single-teacher paradigm fails to impart comprehensive knowledge, The other is the existing methods …
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia（英属哥伦比亚大学）；Fudan University（复旦大学）；Royal College of Science, Imperial College London（伦敦帝国理工学院皇家科学学院）；Dyson School of Design Engineering（戴森设计工程学院）；Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences（中国科学院苏州生物医学工程技术研究所）；East China Normal University（华东师范大学）
