Multi-Scale Global-Instance Prompt Tuning for Continual Test-time Adaptation in Medical Image Segmentation
School of Computer Science, University of Nottingham, UK（诺丁汉大学计算机科学学院）；School of Artificial Intelligence, Shenzhen University, China（深圳大学人工智能学院）；Department of Information Engineering and Computer Science, University of Trento, Italy（特伦托大学信息工程与计算机科学系）；School of Computer Science and Information Engineering, Hefei University of Technology, China（合肥工业大学计算机科学与信息工程学院）
Lingrui Li, Yanfeng Zhou, Nan Pu, Xin Chen, Zhun Zhong
Distribution shift is a common challenge in medical images obtained from different clinical centers, significantly hindering the deployment of pre-trained semantic segmentation models in real-world applications across multiple domains. Continual Test-Time Adaptation(CTTA) has emerged as a promising approach to address cross-domain shifts during continually evolving target domains. Most existing CTTA methods rely on incrementally updating model parameters, which inevitably suffer from error accumulation and catastrophic forgetting, especially in long-term adaptation. Recent prompt-tuning-based works have shown potential to mitigate the two issues above by updating only visual prompts. While these approaches have demonstrated promising performance, several limitations remain:1)lacking multi-scale prompt diversity, 2)inadequate incorporation of instance-specific knowledge, and 3)risk of privacy leakage. To overcome these limitations, we propose Multi-scale Global-Instance Prompt Tuning(MGIPT), to enhance scale diversity of prompts and capture both global- and instance-level knowledge for robust CTTA. Specifically, MGIPT consists of an Adaptive-scale Instance Prompt(AIP) and a Multi-scale Global-level Prompt(MGP). AIP dynamically learns lightweight and instance-specific prompts to mitigate error accumulation with adaptive optimal-scale selection mechanism. MGP captures domain-level knowledge across different scales to ensure robust adaptation with anti-forgetting capabilities. These complementary components are combined through a weighted ensemble approach, enabling effective dual-level adaptation that integrates both global and local information. Extensive experiments on medical image segmentation benchmarks demonstrate that our MGIPT outperforms state-of-the-art methods, achieving robust adaptation across continually changing target domains.
Technische Universität Braunschweig（布伦瑞克工业大学）；Institute of Communications Technology（通信技术研究所）
Environment perception in autonomous driving vehicles often heavily relies on deep neural networks (DNNs), which are subject to domain shifts, leading to a significantly decreased performance during DNN deployment. Usually, this problem is addressed by unsupervised domain adaptation (UDA) approaches trained either simultaneously on source and target domain datasets or even source-free only on target data in an offline fashion. In this work, we further expand a source-free UDA approach to a continual and therefore online-capable UDA on a single-image basis for semantic segmentation. Accordingly…
Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation
Peking University（北京大学）；Institute of Medical Technology, Peking University（北京大学医学技术研究所）；Institute of Biomedical Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院生物医学工程研究所）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）
Continual Test-Time Adaptation (CTTA) generalizes conventional Test-Time Adaptation (TTA) by assuming that the target domain is dynamic over time rather than stationary. In this paper, we explore Multi-Modal Continual Test-Time Adaptation (MM-CTTA) as a new extension of CTTA for 3D semantic segmentation. The key to MM-CTTA is to adaptively attend to the reliable modality while avoiding catastrophic forgetting during continual domain shifts, which is out of the capability of previous TTA or CTTA methods. To fulfill this gap, we propose an MM-CTTA method called Continual Cross-Modal Adaptive Clu…
Technical University of Darmstadt（达姆施塔特工业大学）；MedCom GmbH（MedCom股份有限公司）
Continual learning protocols are attracting increasing attention from the medical imaging community. In continual environments, datasets acquired under different conditions arrive sequentially; and each is only available for a limited period of time. Given the inherent privacy risks associated with medical data, this setup reflects the reality of deployment for deep learning diagnostic radiology systems. Many techniques exist to learn continuously for image classification, and several have been adapted to semantic segmentation. Yet most struggle to accumulate knowledge in a meaningful manner. …
Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation
National Key Laboratory for Multimedia Information Processing, School of CS, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；University of Washington（华盛顿大学）；Baidu Inc.（百度公司）
In clinical settings, where acquisition conditions and patient populations change over time, continual learning is key for ensuring the safe use of deep neural networks. Yet most existing work focuses on convolutional architectures and image classification. Instead, radiologists prefer to work with segmentation models that outline specific regions-of-interest, for which Transformer-based architectures are gaining traction. The self-attention mechanism of Transformers could potentially mitigate catastrophic forgetting, opening the way for more robust medical image segmentation. In this work, we…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）
We develop an algorithm for adapting a semantic segmentation model that is trained using a labeled source domain to generalize well in an unlabeled target domain. A similar problem has been studied extensively in the unsupervised domain adaptation (UDA) literature, but existing UDA algorithms require access to both the source domain labeled data and the target domain unlabeled data for training a domain agnostic semantic segmentation model. Relaxing this constraint enables a user to adapt pretrained models to generalize in a target domain, without requiring access to source data. To this end, …
Most standard learning approaches lead to fragile models which are prone to drift when sequentially trained on samples of a different nature - the well-known "catastrophic forgetting" issue. In particular, when a model consecutively learns from different visual domains, it tends to forget the past domains in favor of the most recent ones. In this context, we show that one way to learn models that are inherently more robust against forgetting is domain randomization - for vision tasks, randomizing the current domain's distribution with heavy image manipulations. Building on this result, we devi…
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Xi’an Jiaotong University（西安交通大学）；School of Information and Communications Engineering, Xi’an Jiaotong University（信息与通信工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院）
Domain Adaptation (DA) techniques are important for overcoming the domain shift between the source domain used for training and the target domain where testing takes place. However, current DA methods assume that the entire target domain is available during adaptation, which may not hold in practice. This paper considers a more realistic scenario, where target data become available in smaller batches and adaptation on the entire target domain is not feasible. In our work, we introduce a new, data-constrained DA paradigm where unlabeled target samples are received in batches and adaptation is p…
Continual Model Merging with Test-Time Adaptation for Whole-Slide Image Analysis
University of Information Technology（信息技术大学）；Viet Nam National University Ho Chi Minh City（胡志明市越南国立大学）；ETIS (UMR 8051), CY Cergy Paris University, ENSEA, CNRS, France（法国赛尔齐巴黎大学、ENSEA、法国国家科学研究中心的ETIS（UMR 8051））
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science（计算机学院）；Peking University（北京大学）；CUHK（香港中文大学）；University of Science Technology of China（中国科学技术大学）
ETH Zurich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）；EPFL（洛桑联邦理工学院）
Test-time domain adaptation aims to adapt a source pre-trained model to a target domain without using any source data. Existing works mainly consider the case where the target domain is static. However, real-world machine perception systems are running in non-stationary and continually changing environments where the target domain distribution can change over time. Existing methods, which are mostly based on self-training and entropy regularization, can suffer from these non-stationary environments. Due to the distribution shift over time in the target domain, pseudo-labels become unreliable. …
Seoul National University（首尔大学）；KAIST（韩国科学技术院）；Kyung Hee University（庆熙大学）
Unsupervised Domain Adaptation (UDA) for semantic segmentation has been favorably applied to real-world scenarios in which pixel-level labels are hard to be obtained. In most of the existing UDA methods, all target data are assumed to be introduced simultaneously. Yet, the data are usually presented sequentially in the real world. Moreover, Continual UDA, which deals with more practical scenarios with multiple target domains in the continual learning setting, has not been actively explored. In this light, we propose Continual UDA for semantic segmentation based on a newly designed Expanding Ta…
Peking University（北京大学）；South China University of Technology（华南理工大学）；National University of Singapore（新加坡国立大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
This paper studies a new, practical but challenging problem, called Class-Incremental Unsupervised Domain Adaptation (CI-UDA), where the labeled source domain contains all classes, but the classes in the unlabeled target domain increase sequentially. This problem is challenging due to two difficulties. First, source and target label sets are inconsistent at each time step, which makes it difficult to conduct accurate domain alignment. Second, previous target classes are unavailable in the current step, resulting in the forgetting of previous knowledge. To address this problem, we propose a nov…
Peking University（北京大学）；Huawei Technologies（华为技术有限公司）；Wuhan University（武汉大学）；The Chinese University of Hong Kong（香港中文大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Continual Test-Time Adaptation (CTTA) aims to adapt the source model to continually changing unlabeled target domains without access to the source data. Existing methods mainly focus on model-based adaptation in a self-training manner, such as predicting pseudo labels for new domain datasets. Since pseudo labels are noisy and unreliable, these methods suffer from catastrophic forgetting and error accumulation when dealing with dynamic data distributions. Motivated by the prompt learning in NLP, in this paper, we propose to learn an image-level visual domain prompt for target domains while havi…
Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS)（蒙特利尔LIVIA ETS，学习系统国际实验室（ILLS））；Tulane University（路易斯安那州立大学）；Nanyang Technological University（南洋理工大学）；Hanyang University（翰阳大学）
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science Peking University（北京大学计算机学院）；AIRobotics
Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen University of Advanced Technology（深圳先进技术学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）
Continual Learning for Segment Anything Model Adaptation
Southern University of Science and Technology（南方科技大学）；City University of Hong Kong（香港城市大学）；Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）
针对动态流式数据场景，提出持续 SAM 适应基准 CoSAM 和域适配器混合算法 MoDA，通过全局特征与辅助令牌提取分离特征，实现持续分割并保持知识，优于现有方法。
Since experiencing domain shifts during test-time is inevitable in practice, test-time adaption (TTA) continues to adapt the model after deployment. Recently, the area of continual and gradual test-time adaptation (TTA) emerged. In contrast to standard TTA, continual TTA considers not only a single domain shift, but a sequence of shifts. Gradual TTA further exploits the property that some shifts evolve gradually over time. Since in both settings long test sequences are present, error accumulation needs to be addressed for methods relying on self-training. In this work, we propose and show that…
Adaptive and Balanced Re-initialization for Long-timescale Continual Test-time Domain Adaptation
Hong Kong Polytechnic University（香港理工大学）；Eastern Institute for Advanced Study（东方高级研究所）；Ant Group（蚂蚁集团）；Australian National University（澳大利亚国立大学）；CSIRO（澳大利亚科学院）；The University of Hong Kong（香港大学）
University of Sydney（悉尼大学）；The Chinese University of Hong Kong（香港中文大学）；Sensetime Group Limited（商汤集团有限公司）
Human beings can quickly adapt to environmental changes by leveraging learning experience. However, adapting deep neural networks to dynamic environments by machine learning algorithms remains a challenge. To better understand this issue, we study the problem of continual domain adaptation, where the model is presented with a labelled source domain and a sequence of unlabelled target domains. The obstacles in this problem are both domain shift and catastrophic forgetting. We propose Gradient Regularized Contrastive Learning (GRCL) to solve the obstacles. At the core of our method, gradient reg…
Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Guangdong Provincial Key Laboratory of Computility Microelectronics（广东省计算微电子重点实验室）
This work investigates continual learning of two segmentation tasks in brain MRI with neural networks. To explore in this context the capabilities of current methods for countering catastrophic forgetting of the first task when a new one is learned, we investigate elastic weight consolidation, a recently proposed method based on Fisher information, originally evaluated on reinforcement learning of Atari games. We use it to sequentially learn segmentation of normal brain structures and then segmentation of white matter lesions. Our findings show this recent method reduces catastrophic forgettin…
Unsupervised Domain Adaptation (UDA) is a transfer learning task which aims at training on an unlabeled target domain by leveraging a labeled source domain. Beyond the traditional scope of UDA with a single source domain and a single target domain, real-world perception systems face a variety of scenarios to handle, from varying lighting conditions to many cities around the world. In this context, UDAs with several domains increase the challenges with the addition of distribution shifts within the different target domains. This work focuses on a novel framework for learning UDA, continuous UDA…
IBM Research Europe（IBM欧洲研究院）；ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）
Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To…
In this paper, our goal is to adapt a pre-trained convolutional neural network to domain shifts at test time. We do so continually with the incoming stream of test batches, without labels. The existing literature mostly operates on artificial shifts obtained via adversarial perturbations of a test image. Motivated by this, we evaluate the state of the art on two realistic and challenging sources of domain shifts, namely contextual and semantic shifts. Contextual shifts correspond to the environment types, for example, a model pre-trained on indoor context has to adapt to the outdoor context on…
