We describe an approach to domain adaptation that is appropriate exactly in the case when one has enough ``target'' data to do slightly better than just using only ``source'' data. Our approach is incredibly simple, easy to implement as a preprocessing step (10 lines of Perl!) and outperforms state-of-the-art approaches on a range of datasets. Moreover, it is trivially extended to a multi-domain adaptation problem, where one has data from a variety of different domains.
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；UC Berkeley（加州大学伯克利分校）
Unlike human learning, machine learning often fails to handle changes between training (source) and test (target) input distributions. Such domain shifts, common in practical scenarios, severely damage the performance of conventional machine learning methods. Supervised domain adaptation methods have been proposed for the case when the target data have labels, including some that perform very well despite being "frustratingly easy" to implement. However, in practice, the target domain is often unlabeled, requiring unsupervised adaptation. We propose a simple, effective, and efficient method fo…
Tencent AI Lab（腾讯人工智能实验室）；Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）
Transferability estimation has been an essential tool in selecting a pre-trained model and the layers in it for transfer learning, to transfer, so as to maximize the performance on a target task and prevent negative transfer. Existing estimation algorithms either require intensive training on target tasks or have difficulties in evaluating the transferability between layers. To this end, we propose a simple, efficient, and effective transferability measure named TransRate. Through a single pass over examples of a target task, TransRate measures the transferability as the mutual information bet…
FIXED: Frustratingly Easy Domain Generalization with Mixup
Tsinghua University（清华大学）；Microsoft（微软）；Nanyang Technological University（南洋理工大学）；University of North Carolina（北卡罗来纳大学）；University of Chinese Academy of Sciences（中国科学院大学）
Mila（米拉研究所）；McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）；UBC Okanagan（不列颠哥伦比亚大学奥卡那根分校）
Distribution shift is an important concern in deep image classification, produced either by corruption of the source images, or a complete change, with the solution involving domain adaptation. While the primary goal is to improve accuracy under distribution shift, an important secondary goal is uncertainty estimation: evaluating the probability that the prediction of a model is correct. While improving accuracy is hard, uncertainty estimation turns out to be frustratingly easy. Prior works have appended uncertainty estimation into the model and training paradigm in various ways. Instead, we s…
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Boston University（波士顿大学）
Deep neural networks are able to learn powerful representations from large quantities of labeled input data, however they cannot always generalize well across changes in input distributions. Domain adaptation algorithms have been proposed to compensate for the degradation in performance due to domain shift. In this paper, we address the case when the target domain is unlabeled, requiring unsupervised adaptation. CORAL is a "frustratingly easy" unsupervised domain adaptation method that aligns the second-order statistics of the source and target distributions with a linear transformation. Here,…
The Hong Kong University of Science and Technology（香港科技大学）；Stanford University（斯坦福大学）；Nanyang Technological University（南洋理工大学）
Learning how to generalize the model to unseen domains is an important area of research. In this paper, we propose TripleE, and the main idea is to encourage the network to focus on training on subsets (learning with replay) and enlarge the data space in learning on subsets. Learning with replay contains two core designs, EReplayB and EReplayD, which conduct the replay schema on batch and dataset, respectively. Through this, the network can focus on learning with subsets instead of visiting the global set at a glance, enlarging the model diversity in ensembling. To enlarge the data space in le…
Frustratingly Easy Task-aware Pruning for Large Language Models
Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；University of Science and Technology of China（中国科学技术大学）；Xi’an Jiaotong University（西安交通大学）
Tencent AI Lab（腾讯人工智能实验室）；Xidian University（西安电子科技大学）；Northwestern Polytechnical University（西北工业大学）；Hong Kong Polytechnic University（香港理工大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Domain adaptation problems arise in a variety of applications, where a training dataset from the \textit{source} domain and a test dataset from the \textit{target} domain typically follow different distributions. The primary difficulty in designing effective learning models to solve such problems lies in how to bridge the gap between the source and target distributions. In this paper, we provide comprehensive analysis of feature learning algorithms used in conjunction with linear classifiers for domain adaptation. Our analysis shows that in order to achieve good adaptation performance, the sec…
Weight Scope Alignment: A Frustratingly Easy Method for Model Merging
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Frustratingly Easy Feature Reconstruction for Out-of-Distribution Detection
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China（MAIS与NLPR、自动化研究所、中国科学院、北京、中国）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（人工智能学院、中国科学院大学、北京、中国）；Anhui University, Hefei, China（安徽大学、合肥、中国）
DataNova；U.S. Army Research Laboratory（美国陆军研究实验室）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Electroencephalography (EEG) headsets are the most commonly used sensing devices for Brain-Computer Interface. In real-world applications, there are advantages to extrapolating data from one user session to another. However, these advantages are limited if the data arise from different hardware systems, which often vary between application spaces. Currently, this creates a need to recalibrate classifiers, which negatively affects people's interest in using such systems. In this paper, we employ active weighted adaptation regularization (AwAR), which integrates weighted adaptation regularizatio…
Convolutional neural network-based approaches have achieved remarkable progress in semantic segmentation. However, these approaches heavily rely on annotated data which are labor intensive. To cope with this limitation, automatically annotated data generated from graphic engines are used to train segmentation models. However, the models trained from synthetic data are difficult to transfer to real images. To tackle this issue, previous works have considered directly adapting models from the source data to the unlabeled target data (to reduce the inter-domain gap). Nonetheless, these techniques…
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）；Tsinghua University（清华大学）
Domain Adaptation of Black-box Predictors (DABP) aims to learn a model on an unlabeled target domain supervised by a black-box predictor trained on a source domain. It does not require access to both the source-domain data and the predictor parameters, thus addressing the data privacy and portability issues of standard domain adaptation. Existing DABP approaches mostly rely on model distillation from the black-box predictor, \emph{i.e.}, training the model with its noisy target-domain predictions, which however inevitably introduces the confirmation bias accumulated from the prediction noises.…
University of Central Florida（中佛罗里达大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is one of the core tasks in many applications such as autonomous driving. However, to train CNNs requires a considerable amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNNs on photo-realistic synthetic imagery with computer-generated annotations. Despite this, the domain mismatch between the real images and the synthetic data cripples the models' performance. Hence, we propose a curriculum-sty…
Cross-domain object detection is more challenging than object classification since multiple objects exist in an image and the location of each object is unknown in the unlabeled target domain. As a result, when we adapt features of different objects to enhance the transferability of the detector, the features of the foreground and the background are easy to be confused, which may hurt the discriminability of the detector. Besides, previous methods focused on category adaptation but ignored another important part for object detection, i.e., the adaptation on bounding box regression. To this end…
Beckman Institute for Advanced Science and Technology（贝克曼高等科学技术研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Adobe Research（Adobe研究院）
We address a challenging fine-grain classification problem: recognizing a font style from an image of text. In this task, it is very easy to generate lots of rendered font examples but very hard to obtain real-world labeled images. This real-to-synthetic domain gap caused poor generalization to new real data in previous methods (Chen et al. (2014)). In this paper, we refer to Convolutional Neural Networks, and use an adaptation technique based on a Stacked Convolutional Auto-Encoder that exploits unlabeled real-world images combined with synthetic data. The proposed method achieves an accuracy…
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data. Like much of previous work, we seek to align the learned representations of the source and target domains while preserving discriminability. The way we accomplish alignment is by learning to perform auxiliary self-supervised task(s) on both domains simultaneously. Each self-supervised task brings the two domains closer together along the direction relevant to that task. Training this j…
Learning Fair Domain Adaptation with Virtual Label Distribution
1 School of Artificial Intelligence, University of Chinese Academy of Sciences 2 NLPR \& MAIS, Institute of Automation, Chinese Academy of Sciences 3 University of Science
Domain adaptation (DA) is the task of classifying an unlabeled dataset (target) using a labeled dataset (source) from a related domain. The majority of successful DA methods try to directly match the distributions of the source and target data by transforming the feature space. Despite their success, state of the art methods based on this approach are either involved or unable to directly scale to data with many features. This article shows that domain adaptation can be successfully performed by using a very simple randomized expectation maximization (EM) method. We consider two instances of t…
Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）
Deep networks are prone to performance degradation when there is a domain shift between the source (training) data and target (test) data. Recent test-time adaptation methods update batch normalization layers of pre-trained source models deployed in new target environments with streaming data to mitigate such performance degradation. Although such methods can adapt on-the-fly without first collecting a large target domain dataset, their performance is dependent on streaming conditions such as mini-batch size and class-distribution, which can be unpredictable in practice. In this work, we propo…
Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（RUDN大学））；Joint Institute for Nuclear Research（联合核研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）；University of Pisa（比萨大学）
Predictive machine learning models nowadays are often updated in a stateless and expensive way. The two main future trends for companies that want to build machine learning-based applications and systems are real-time inference and continual updating. Unfortunately, both trends require a mature infrastructure that is hard and costly to realize on-premise. This paper defines a novel software service and model delivery infrastructure termed Continual Learning-as-a-Service (CLaaS) to address these issues. Specifically, it embraces continual machine learning and continuous integration techniques. …
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation
NSF AI Institute for Artificial Intelligence and Fundamental Interactions（NSF人工智能与基础交互AI研究所）；Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）
Learning Invariant Representations has been successfully applied for reconciling a source and a target domain for Unsupervised Domain Adaptation. By investigating the robustness of such methods under the prism of the cluster assumption, we bring new evidence that invariance with a low source risk does not guarantee a well-performing target classifier. More precisely, we show that the cluster assumption is violated in the target domain despite being maintained in the source domain, indicating a lack of robustness of the target classifier. To address this problem, we demonstrate the importance o…
Air quality monitoring is becoming an essential task with rising awareness about air quality. Low cost air quality sensors are easy to deploy but are not as reliable as the costly and bulky reference monitors. The low quality sensors can be calibrated against the reference monitors with the help of deep learning. In this paper, we translate the task of sensor calibration into a semi-supervised domain adaptation problem and propose a novel solution for the same. The problem is challenging because it is a regression problem with covariate shift and label gap. We use histogram loss instead of mea…
Unsupervised Domain Adaptation (DA) is used to automatize the task of labeling data: an unlabeled dataset (target) is annotated using a labeled dataset (source) from a related domain. We cast domain adaptation as the problem of finding stable labels for target examples. A new definition of label stability is proposed, motivated by a generalization error bound for large margin linear classifiers: a target labeling is stable when, with high probability, a classifier trained on a random subsample of the target with that labeling yields the same labeling. We find stable labelings using a random wa…
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）；China Academy of Space Technology（中国空间技术研究院）
The recent advances in deep transfer learning reveal that adversarial learning can be embedded into deep networks to learn more transferable features to reduce the distribution discrepancy between two domains. Existing adversarial domain adaptation methods either learn a single domain discriminator to align the global source and target distributions or pay attention to align subdomains based on multiple discriminators. However, in real applications, the marginal (global) and conditional (local) distributions between domains are often contributing differently to the adaptation. There is current…
Unsupervised domain adaptation (UDA) transfers knowledge from a label-rich source domain to a fully-unlabeled target domain. To tackle this task, recent approaches resort to discriminative domain transfer in virtue of pseudo-labels to enforce the class-level distribution alignment across the source and target domains. These methods, however, are vulnerable to the error accumulation and thus incapable of preserving cross-domain category consistency, as the pseudo-labeling accuracy is not guaranteed explicitly. In this paper, we propose the Progressive Feature Alignment Network (PFAN) to align t…
The University of Hong Kong（香港大学）；CUHK-SenseTime Joint Laboratory（香港中文大学-商汤科技联合实验室）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；School of CST Xidian University（西安电子科技大学计算机科学与技术学院）
We present a new domain adaptive self-training pipeline, named ST3D, for unsupervised domain adaptation on 3D object detection from point clouds. First, we pre-train the 3D detector on the source domain with our proposed random object scaling strategy for mitigating the negative effects of source domain bias. Then, the detector is iteratively improved on the target domain by alternatively conducting two steps, which are the pseudo label updating with the developed quality-aware triplet memory bank and the model training with curriculum data augmentation. These specific designs for 3D object de…
South China University of Technology（华南理工大学）；Hong Kong Polytechnic University（香港理工大学）
In this paper, we study an arguably least restrictive setting of domain adaptation in a sense of practical deployment, where only the interface of source model is available to the target domain, and where the label-space relations between the two domains are allowed to be different and unknown. We term such a setting as Universal Black-Box Domain Adaptation (UB$^2$DA). The great promise that UB$^2$DA makes, however, brings significant learning challenges, since domain adaptation can only rely on the predictions of unlabeled target data in a partially overlapped label space, by accessing the in…
Dalian University of Technology（大连理工大学）；Baidu Research（百度研究院）；Tulane University（杜兰大学）；University of Shenzhen（深圳大学）；University of Queensland（昆士兰大学）
Domain adaptation has received a lot of attention in recent years, and many algorithms have been proposed with impressive progress. However, it is still not fully explored concerning the joint probability distribution (P(X, Y)) distance for this problem, since its empirical estimation derived from the maximum mean discrepancy (joint maximum mean discrepancy, JMMD) will involve complex tensor-product operator that is hard to manipulate. To solve this issue, this paper theoretically derives a unified form of JMMD that is easy to optimize, and proves that the marginal, class conditional and weigh…
NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Youtu Lab, Tencent Inc.（腾讯优图实验室）；CASIA-LLvision Joint Lab（中国科学院自动化研究所与联影智能联合实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳市大数据研究院）
Big progress has been achieved in domain adaptation in decades. Existing works are always based on an ideal assumption that testing target domain are i.i.d. with training target domains. However, due to unpredictable corruptions (e.g., noise and blur) in real data like web images, domain adaptation methods are increasingly required to be corruption robust on target domains. In this paper, we investigate a new task, Corruption-agnostic Robust Domain Adaptation (CRDA): to be accurate on original data and robust against unavailable-for-training corruptions on target domains. This task is non-triv…
Dalhousie University（达尔豪斯大学）；Imagia Cybernetics（Imagia控制论公司）；Polish Academy of Sciences（波兰科学院）
Transferring knowledge from one neural network to another has been shown to be helpful for learning tasks with few training examples. Prevailing fine-tuning methods could potentially contaminate pre-trained features by comparably high energy random noise. This noise is mainly delivered from a careless replacement of task-specific parameters. We analyze theoretically such knowledge contamination for classification tasks and propose a practical and easy to apply method to trap and minimize the contaminant. In our approach, the entropy of the output estimates gets maximized initially and the firs…
University of Pennsylvania（宾夕法尼亚大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
Domain shift, the mismatch between training and testing data characteristics, causes significant degradation in the predictive performance in multi-source imaging scenarios. In medical imaging, the heterogeneity of population, scanners and acquisition protocols at different sites presents a significant domain shift challenge and has limited the widespread clinical adoption of machine learning models. Harmonization methods which aim to learn a representation of data invariant to these differences are the prevalent tools to address domain shift, but they typically result in degradation of predic…
Data-efficient domain adaptation with only a few labelled data is desired for many robotic applications, e.g., in grasping detection, the inference skill learned from a grasping dataset is not universal enough to directly apply on various other daily/industrial applications. This paper presents an approach enabling the easy domain adaptation through a novel grasping detection network with confidence-driven semi-supervised learning, where these two components deeply interact with each other. The proposed grasping detection network specially provides a prediction uncertainty estimation mechanism…
Cornell University（康奈尔大学）；The Ohio State University（俄亥俄州立大学）
For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment -- ideally without additional annotation efforts. One potential solution is to leverage unlabeled data (e.g., unlabeled LiDAR point clouds) collected from the end-users' environments (i.e. target domain) to adapt the system to the difference between training and testing environments. While extensive research has been done on such an unsupervised domain adaptation problem, one fundamental problem lingers: there is no reliable signal in the target domain to supervise the adaptation pro…
Training deep networks for semantic segmentation requires annotation of large amounts of data, which can be time-consuming and expensive. Unfortunately, these trained networks still generalize poorly when tested in domains not consistent with the training data. In this paper, we show that by carefully presenting a mixture of labeled source domain and proxy-labeled target domain data to a network, we can achieve state-of-the-art unsupervised domain adaptation results. With our design, the network progressively learns features specific to the target domain using annotation from only the source d…
SelfAdapt: Unsupervised Domain Adaptation of Cell Segmentation Models
Charité - Universitätsmedizin（Charité - 大学医院）；Humboldt-Universität zu Berlin（洪堡-柏林大学）；Universität Potsdam（波茨坦大学）；Helmholtz Imaging（海德堡成像）；Max Delbrück Center for Molecular Medicine in the Helmholtz Association（海德堡协会分子医学中心）；Ludwig-Maximilians-University（路德维希-马克西米利安大学）
A common assumption in semi-supervised learning with graph models is that the class label function varies smoothly on the data graph, resulting in the rather strict prior that the label function has low-frequency content. Meanwhile, in many classification problems, the label function may vary abruptly in certain graph regions, resulting in high-frequency components. Although the semi-supervised estimation of class labels is an ill-posed problem in general, in several applications it is possible to find a source graph on which the label function has similar frequency content to that on the targ…
University of Maryland（马里兰大学）；Cornell University（康奈尔大学）；Salesforce Research（Salesforce研究院）；Ohio State University（俄亥俄州立大学）；Facebook AI（Facebook人工智能）
Semi-supervised domain adaptation (SSDA) aims to adapt models trained from a labeled source domain to a different but related target domain, from which unlabeled data and a small set of labeled data are provided. Current methods that treat source and target supervision without distinction overlook their inherent discrepancy, resulting in a source-dominated model that has not effectively used the target supervision. In this paper, we argue that the labeled target data needs to be distinguished for effective SSDA, and propose to explicitly decompose the SSDA task into two sub-tasks: a semi-super…
PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation
Westlake Institute for Advanced Studies（西湖研究学院）；Centre for Artificial Intelligence（人工智能中心）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation（自动化研究所）；Chinese Academy of Sciences (CASIA)（中国科学院）；School of Artificial Intelligence（人工智能学院）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Alibaba Group（阿里巴巴集团）；School of Computer Science（计算机学院）；Engineering, the Faculty of Innovation Engineering, Macau University of Science（工程学院，创新工程 faculty，澳门科学大学）；Microsoft（微软公司）
University of California, Berkeley（加州大学伯克利分校）；OpenAI
We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new ta…
MixStyle Neural Networks for Domain Generalization and Adaptation
Queen Mary University of London（伦敦玛丽女王大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；University of Surrey（萨里大学）；Hong Kong Baptist University（香港浸会大学）
