ETAGE: Enhanced Test Time Adaptation with Integrated Entropy and Gradient Norms for Robust Model Performance
Concordia University（康考迪亚大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；UNSW Sydney（新南威尔士大学悉尼分校）；Concordia Institute of Information Systems Engineering（康考迪亚信息系统工程学院）；The Graduate School of Biomedical Engineering（生物医学工程研究生院）；School of Biomedical Engineering（生物医学工程学院）
测试时自适应（TTA）使深度学习模型即使在无法获取源数据的情况下，也能处理偏离训练分布的未见过的测试数据。传统TTA方法通常依赖熵作为置信度指标，但其有效性可能受限，尤其在有偏场景中。在伪标签概率差（Pseudo Label Probability Difference, PLPD）等现有方法的基础上，我们提出了ETAGE，这是一种改进的TTA方法，将熵最小化与梯度范数、PLPD相结合，以优化样本选择和自适应过程。我们的方法通过将高熵与自适应过程外的高梯度范数相结合，优先选择不太可能导致不稳定的样本，从而避免了以往方法中常见的对噪声的过拟合问题。在CIFAR-10-C和CIFAR-100-C数据集上开展的大量实验表明，我们的方法优于现有TTA技术，尤其在具有挑战性的有偏场景中表现更优，能在多样测试场景下带来更鲁棒、更一致的模型性能。ETAGE的代码库可在https://github.com/afsharshamsi/ETAGE获取。
英文摘要：
Test time adaptation (TTA) equips deep learning models to handle unseen test data that deviates from the training distribution, even when source data is inaccessible. While traditional TTA methods often rely on entropy as a confidence metric, its effectiveness can be limited, particularly in biased scenarios. Extending existing approaches like the Pseudo Label Probability Difference (PLPD), we introduce ETAGE, a refined TTA method that integrates entropy minimization with gradient norms and PLPD, to enhance sample selection and adaptation. Our method prioritizes samples that are less likely to cause instability by combining high entropy with high gradient norms out of adaptation, thus avoiding the overfitting to noise often observed in previous methods. Extensive experiments on CIFAR-10-C and CIFAR-100-C datasets demonstrate that our approach outperforms existing TTA techniques, particularly in challenging and biased scenarios, leading to more robust and consistent model performance across diverse test scenarios. The codebase for ETAGE is available on https://github.com/afsharshamsi/ETAGE.
We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a two-stage algorithm. In the first stage, we estimate pseudo-labels for the examples in the target domain using an external unsupervised domain adaptation algorithm---for example, MSTN or CPUA---integrating the proposed domain-specific batch normalization. The second stage learns th…
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney（澳大利亚机器人中心，航空航天、机械与机电工程学院，悉尼大学）；Google DeepMind（谷歌DeepMind）
In this paper, we propose a novel domain adaptation method for the source-free setting. In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given. Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms. To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data. Specifically, we fix the classifier part of the model during a…
Blue Well 8, LLC（Blue Well 8有限责任公司）；Pioneer Square Labs（先锋广场实验室）
Adapting a model to perform well on unforeseen data outside its training set is a common problem that continues to motivate new approaches. We demonstrate that application of batch normalization in the output layer, prior to softmax activation, results in improved generalization across visual data domains in a refined ResNet model. The approach adds negligible computational complexity yet outperforms many domain adaptation methods that explicitly learn to align data domains. We benchmark this technique on the Office-Home dataset and show that batch normalization is competitive with other leadi…
Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al. 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called A…
Understanding proper distance measures between distributions is at the core of several learning tasks such as generative models, domain adaptation, clustering, etc. In this work, we focus on mixture distributions that arise naturally in several application domains where the data contains different sub-populations. For mixture distributions, established distance measures such as the Wasserstein distance do not take into account imbalanced mixture proportions. Thus, even if two mixture distributions have identical mixture components but different mixture proportions, the Wasserstein distance bet…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Unsupervised Source (data) Free domain adaptation (USFDA) aims to transfer knowledge from a well-trained source model to a related but unlabeled target domain. In such a scenario, all conventional adaptation methods that require source data fail. To combat this challenge, existing USFDAs turn to transfer knowledge by aligning the target feature to the latent distribution hidden in the source model. However, such information is naturally limited. Thus, the alignment in such a scenario is not only difficult but also insufficient, which degrades the target generalization performance. To relieve t…
School of Electrical and Electronic Engineering, Yonsei University（延世大学电气电子工程学院）；Department of Radiology and Center for Clinical Imaging Data Science, College of Medicine, Yonsei University（延世大学医学院放射科和临床影像数据科学中心）；Artificial Intelligence and Robotics Institute, Korea Institute of Science and Technology（韩国科学技术院人工智能与机器人研究所）
pFedBBN: A Personalized Federated Test-Time Adaptation with Balanced Batch Normalization for Class-Imbalanced Data
Center for Computational & Data Sciences (CCDS)（计算与数据科学中心）；Independent University（独立大学）；University of Central Florida（中央佛罗里达大学）；Curtin University（Curtin大学）；Umeå University（乌梅拉大学）
For a long time, anomaly localization has been widely used in industries. Previous studies focused on approximating the distribution of normal features without adaptation to a target dataset. However, since anomaly localization should precisely discriminate normal and abnormal features, the absence of adaptation may make the normality of abnormal features overestimated. Thus, we propose Coupled-hypersphere-based Feature Adaptation (CFA) which accomplishes sophisticated anomaly localization using features adapted to the target dataset. CFA consists of (1) a learnable patch descriptor that learn…
Durham University（杜伦大学）；University of Essex（埃塞克斯大学）
We address the Unsupervised Domain Adaptation (UDA) problem in image classification from a new perspective. In contrast to most existing works which either align the data distributions or learn domain-invariant features, we directly learn a unified classifier for both domains within a high-dimensional homogeneous feature space without explicit domain adaptation. To this end, we employ the effective Selective Pseudo-Labelling (SPL) techniques to take advantage of the unlabelled samples in the target domain. Surprisingly, data distribution discrepancy across the source and target domains can be …
Univ Lyon（里昂大学）；Ecole Centrale de Lyon（里昂中央理工学院）；CNRS（法国国家科学研究中心）；INSA Lyon（里昂国立应用科学学院）；Univ Claude Bernard Lyon 1（里昂第一大学克洛德·贝尔纳）；Univ Louis Lumière Lyon 2（里昂第二大学路易·卢米埃尔）；Thales LAS France（泰雷兹LAS法国公司）
Deep neural networks often fail to generalize outside of their training distribution, in particular when only a single data domain is available during training. While test-time adaptation has yielded encouraging results in this setting, we argue that, to reach further improvements, these approaches should be combined with training procedure modifications aiming to learn a more diverse set of patterns. Indeed, test-time adaptation methods usually have to rely on a limited representation because of the shortcut learning phenomenon: only a subset of the available predictive patterns is learned wi…
LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devices
Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；University of Cambridge（剑桥大学）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）；Samsung AI Center（三星人工智能中心）
We study generalization under labeled shift for categorical and general normed label spaces. We propose a series of methods to estimate the importance weights from labeled source to unlabeled target domain and provide confidence bounds for these estimators. We deploy these estimators and provide generalization bounds in the unlabeled target domain.
Tsinghua University（清华大学）；Tencent TEG AI（腾讯技术工程事业群人工智能实验室）；OPPO research institute（OPPO研究院）；Peng Cheng Laboratory（鹏城实验室）
Fully test-time adaptation aims at adapting a pre-trained model to the test stream during real-time inference, which is urgently required when the test distribution differs from the training distribution. Several efforts have been devoted to improving adaptation performance. However, we find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimate…
RIKEN Center for Advanced Intelligence Project (RIKEN AIP)（理化学研究所先进智能项目中心）；Advanced Telecommunications Research Institute International (ATR)（国际电气通信基础技术研究所）
Electroencephalography (EEG) provides access to neuronal dynamics non-invasively with millisecond resolution, rendering it a viable method in neuroscience and healthcare. However, its utility is limited as current EEG technology does not generalize well across domains (i.e., sessions and subjects) without expensive supervised re-calibration. Contemporary methods cast this transfer learning (TL) problem as a multi-source/-target unsupervised domain adaptation (UDA) problem and address it with deep learning or shallow, Riemannian geometry aware alignment methods. Both directions have, so far, fa…
Unsupervised Domain Adaptation (UDA) aims to transfer the knowledge from the labeled source domain to the unlabeled target domain in the presence of dataset shift. Most existing methods cannot address the domain alignment and class discrimination well, which may distort the intrinsic data structure for downstream tasks (e.g., classification). To this end, we propose a novel geometry-aware model to learn the transferability and discriminability simultaneously via nuclear norm optimization. We introduce the domain coherence and class orthogonality for UDA from the perspective of subspace geometr…
Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalization
South China University of Technology（华南理工大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Test-time adaptation (TTA) is an emerging paradigm that addresses distributional shifts between training and testing phases without additional data acquisition or labeling cost; only unlabeled test data streams are used for continual model adaptation. Previous TTA schemes assume that the test samples are independent and identically distributed (i.i.d.), even though they are often temporally correlated (non-i.i.d.) in application scenarios, e.g., autonomous driving. We discover that most existing TTA methods fail dramatically under such scenarios. Motivated by this, we present a new test-time a…
Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）
Deep networks are prone to performance degradation when there is a domain shift between the source (training) data and target (test) data. Recent test-time adaptation methods update batch normalization layers of pre-trained source models deployed in new target environments with streaming data to mitigate such performance degradation. Although such methods can adapt on-the-fly without first collecting a large target domain dataset, their performance is dependent on streaming conditions such as mini-batch size and class-distribution, which can be unpredictable in practice. In this work, we propo…
We study the problem of semi-supervised anomaly detection with domain adaptation. Given a set of normal data from a source domain and a limited amount of normal examples from a target domain, the goal is to have a well-performing anomaly detector in the target domain. We propose the Invariant Representation Anomaly Detection (IRAD) to solve this problem where we first learn to extract a domain-invariant representation. The extraction is achieved by an across-domain encoder trained together with source-specific encoders and generators by adversarial learning. An anomaly detector is then trained…
Many meta-learning algorithms can be formulated into an interleaved process, in the sense that task-specific predictors are learned during inner-task adaptation and meta-parameters are updated during meta-update. The normal meta-training strategy needs to differentiate through the inner-task adaptation procedure to optimize the meta-parameters. This leads to a constraint that the inner-task algorithms should be solved analytically. Under this constraint, only simple algorithms with analytical solutions can be applied as the inner-task algorithms, limiting the model expressiveness. To lift the …
This work introduces the novel task of Source-free Multi-target Domain Adaptation and proposes adaptation framework comprising of \textbf{Co}nsistency with \textbf{N}uclear-Norm Maximization and \textbf{Mix}Up knowledge distillation (\textit{CoNMix}) as a solution to this problem. The main motive of this work is to solve for Single and Multi target Domain Adaptation (SMTDA) for the source-free paradigm, which enforces a constraint where the labeled source data is not available during target adaptation due to various privacy-related restrictions on data sharing. The source-free approach leverag…
Recent progress of self-supervised visual representation learning has achieved remarkable success on many challenging computer vision benchmarks. However, whether these techniques can be used for domain adaptation has not been explored. In this work, we propose a generic method for self-supervised domain adaptation, using object recognition and semantic segmentation of urban scenes as use cases. Focusing on simple pretext/auxiliary tasks (e.g. image rotation prediction), we assess different learning strategies to improve domain adaptation effectiveness by self-supervision. Additionally, we pro…
SNAP: Low-Latency Test-Time Adaptation with Sparse Updates
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea（韩国科学技术院电子工程学院）；Department of Computer Science and Engineering, UNIST, Ulsan, Republic of Korea（全南国立大学计算机科学与工程系）
When Unsupervised Domain Adaptation meets One-class Anomaly Detection: Addressing the Two-fold Unsupervised Curse by Leveraging Anomaly Scarcity
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（安全、可靠性与信任跨学科研究中心（SnT），卢森堡大学）；Cristal Laboratory, National School of Computer Sciences, University of Manouba（晶体实验室，国家计算机科学学校，曼努巴大学）
Efficiently adapting to new environments and changes in dynamics is critical for agents to successfully operate in the real world. Reinforcement learning (RL) based approaches typically rely on external reward feedback for adaptation. However, in many scenarios this reward signal might not be readily available for the target task, or the difference between the environments can be implicit and only observable from the dynamics. To this end, we introduce a method that allows for self-adaptation of learned policies: No-Reward Meta Learning (NoRML). NoRML extends Model Agnostic Meta Learning (MAML…
Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
Poznan University of Technology（波兹南理工大学）；IDEAS NCBR；NASK - National Research Institute（NASK国家研究所）；Gdańsk University of Technology（格但斯克理工大学）；Graz University of Technology（格拉茨理工大学）
South China University of Technology（华南理工大学）；Tencent AI Lab（腾讯人工智能实验室）；National University of Singapore（新加坡国立大学）；Pazhou Laboratory（琶洲实验室）
Test-time adaptation (TTA) has shown to be effective at tackling distribution shifts between training and testing data by adapting a given model on test samples. However, the online model updating of TTA may be unstable and this is often a key obstacle preventing existing TTA methods from being deployed in the real world. Specifically, TTA may fail to improve or even harm the model performance when test data have: 1) mixed distribution shifts, 2) small batch sizes, and 3) online imbalanced label distribution shifts, which are quite common in practice. In this paper, we investigate the unstable…
Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models
Northwestern University（西北大学）；University of Science and Technology of China（中国科学技术大学）；National Yang Ming Chiao Tung University（阳明交通大学）；Simons Institute UC Berkeley（伯克利Simons研究所）
International Multi-Centre Validation of Unsupervised Domain Adaptation for Precise Discrimination between Normal and Abnormal Retinal Development
The University of Leicester（莱斯特大学）；International Institute of Information Technology（国际信息技术学院）；University of Leicester（莱斯特大学）；University of Sheffield（谢菲尔德大学）；Copenhagen University Hospital, Rigshospitalet-Glostrup（哥本哈根大学医院，rigshospitalet-glostrup院区）；University of Copenhagen（哥本哈根大学）；Yonsei University College of Medicine（延世大学医学院）
Fine-tuning neural networks is widely used to transfer valuable knowledge from high-resource to low-resource domains. In a standard fine-tuning scheme, source and target problems are trained using the same architecture. Although capable of adapting to new domains, pre-trained units struggle with learning uncommon target-specific patterns. In this paper, we propose to augment the target-network with normalised, weighted and randomly initialised units that beget a better adaptation while maintaining the valuable source knowledge. Our experiments on POS tagging of social media texts (Tweets domai…
