Reliable uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and leverage predictive models. However, existing approaches to estimating uncertainty largely ignore the possibility of covariate shift--i.e., where the real-world data distribution may differ from the training distribution. As a consequence, existing algorithms can overestimate certainty, possibly yielding a false sense of confidence in the predictive model. We propose an algorithm for calibrating predictions that accounts for the possibility of covariate shift, given labeled examples from the training distribution and unlabeled examples from the real-world distribution. Our algorithm uses importance weighting to correct for the shift from the training to the real-world distribution. However, importance weighting relies on the training and real-world distributions to be sufficiently close. Building on ideas from domain adaptation, we additionally learn a feature map that tries to equalize these two distributions. In an empirical evaluation, we show that our proposed approach outperforms existing approaches to calibrated prediction when there is covariate shift.
Unsupervised Domain Adaptation aims to learn a model on a source domain with labeled data in order to perform well on unlabeled data of a target domain. Current approaches focus on learning \textit{Domain Invariant Representations}. It relies on the assumption that such representations are well-suited for learning the supervised task in the target domain. We rather believe that a better and minimal assumption for performing Domain Adaptation is the \textit{Hidden Covariate Shift} hypothesis. Such approach consists in learning a representation of the data such that the label distribution condit…
Massachusetts Institute of Technology（麻省理工学院）；Rutgers University（罗格斯大学）
Existing domain adaptation (DA) usually assumes the domain shift comes from either the covariates or the labels. However, in real-world applications, samples selected from different domains could have biases in both the covariates and the labels. In this paper, we propose a new assumption, Factorizable Joint Shift (FJS), to handle the co-existence of sampling bias in covariates and labels. Although allowing for the shift from both sides, FJS assumes the independence of the bias between the two factors. We provide theoretical and empirical understandings about when FJS degenerates to prior assu…
TILT: Target-induced loss tilting under covariate shift
Lab for Information and Decision Systems（信息与决策系统实验室）；Statistics and Data Science Center（统计与数据科学中心）；EECS, Massachusetts Institute of Technology（麻省理工学院电子工程与计算机科学系）；Mathematics and EECS, Massachusetts Institute of Technology（数学与电子工程与计算机科学系, 麻省理工学院）
Air quality monitoring is becoming an essential task with rising awareness about air quality. Low cost air quality sensors are easy to deploy but are not as reliable as the costly and bulky reference monitors. The low quality sensors can be calibrated against the reference monitors with the help of deep learning. In this paper, we translate the task of sensor calibration into a semi-supervised domain adaptation problem and propose a novel solution for the same. The problem is challenging because it is a regression problem with covariate shift and label gap. We use histogram loss instead of mea…
A probabilistic model is said to be calibrated if its predicted probabilities match the corresponding empirical frequencies. Calibration is important for uncertainty quantification and decision making in safety-critical applications. While calibration of classifiers has been widely studied, we find that calibration is brittle and can be easily lost under minimal covariate shifts. Existing techniques, including domain adaptation ones, primarily focus on prediction accuracy and do not guarantee calibration neither in theory nor in practice. In this work, we formally introduce the problem of cali…
Nanyang Technological University（南洋理工大学）；Singapore Institute of Manufacturing Technology（新加坡制造技术研究院）
We consider the problem of online unsupervised cross-domain adaptation, where two independent but related data streams with different feature spaces -- a fully labeled source stream and an unlabeled target stream -- are learned together. Unique characteristics and challenges such as covariate shift, asynchronous concept drifts, and contrasting data throughput arises. We propose ACDC, an adversarial unsupervised domain adaptation framework that handles multiple data streams with a complete self-evolving neural network structure that reacts to these defiances. ACDC encapsulates three modules int…
SRI International（SRI国际）；School of Electrical Engineering and Computer Science, Oregon State University（俄勒冈州立大学电气工程与计算机科学学院）
Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation. However, these methods suffer from the following limitations: 1) they require unlabeled data from the target domain, which may not be available at the stage of calibration in real-world applications and 2) their performance depends heavily on the disparity between the distributions of the source and target domains. To address these two limitations, we present novel calibration solutions via domain generalization. Our core idea is to leverage multiple calibration domains to reduce the effe…
Distributionally Robust Transfer Learning with Structurally Missing Covariates, with Application to Cross-National Cardiac Arrest Prediction
Centre for Biomedical Data Science, Duke-NUS Medical School, Singapore（生物医学数据科学中心，杜克-国家大学医学院，新加坡）；Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore（杜克-国家大学医学院AI+医学科学倡议，新加坡）；Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, USA（生物统计学与生物信息学系，杜克大学，北卡罗来纳州达勒姆，美国）；Duke Clinical Research Institute, Durham, NC, USA（杜克临床研究学院，北卡罗来纳州达勒姆，美国）；Emergency Medicine Department, National University Hospital, Singapore（急诊医学部，国立大学医院，新加坡）；Department of Sport and Medical Science, Faculty of Physical Education, Kokushikan University, Tokyo, Japan（体育与医学科学系，体育学院，立命馆大学，东京，日本）；Graduate School of Emergency Medical System, Kokushikan University, Tokyo, Japan（急救医疗系统研究生院，立命馆大学，东京，日本）；Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea（急诊医学系，首尔国立大学医学院，首尔，韩国）；Center for Emergency Medicine, Bach Mai Hospital, Hanoi, Vietnam（急救医学中心，巴赫梅医院，河内，越南）；Center for Critical Care Medicine, Bach Mai Hospital, Hanoi, Vietnam（重症医学中心，巴赫梅医院，河内，越南）；Health Services Research Centre, Singapore Health Services, Singapore（卫生服务研究中心，新加坡卫生服务，新加坡）；Department of Emergency Medicine, Singapore General Hospital, Singapore（急诊医学部，新加坡中央医院，新加坡）；Pre-hospital & Emergency Research Centre, Health Services Research and Population Health, Duke-NUS Medical School, Singapore（院前与急诊研究中心，卫生服务研究与人口健康，杜克-国家大学医学院，新加坡）
Google Research（谷歌研究院）；Deepmind（深度思维）；Stanford University（斯坦福大学）；Gatsby Computational Neuroscience Unit（盖茨比计算神经科学中心）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We address the problem of unsupervised domain adaptation when the source domain differs from the target domain because of a shift in the distribution of a latent subgroup. When this subgroup confounds all observed data, neither covariate shift nor label shift assumptions apply. We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain, and unlabeled data from the target. The identification results are constructive, immediately suggesting an algorithm for estimating the optimal predictor in the…
Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models
School of Mathematics and Computer Science, Institute of Business Administration（商业管理学院数学与计算机科学学院）；Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）；National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）
We propose a hypothesis disparity regularized mutual information maximization~(HDMI) approach to tackle unsupervised hypothesis transfer -- as an effort towards unifying hypothesis transfer learning (HTL) and unsupervised domain adaptation (UDA) -- where the knowledge from a source domain is transferred solely through hypotheses and adapted to the target domain in an unsupervised manner. In contrast to the prevalent HTL and UDA approaches that typically use a single hypothesis, HDMI employs multiple hypotheses to leverage the underlying distributions of the source and target hypotheses. To bet…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Harvard University（哈佛大学）
In this paper, we propose Normality-Calibrated Autoencoder (NCAE), which can boost anomaly detection performance on the contaminated datasets without any prior information or explicit abnormal samples in the training phase. The NCAE adversarially generates high confident normal samples from a latent space having low entropy and leverages them to predict abnormal samples in a training dataset. NCAE is trained to minimise reconstruction errors in uncontaminated samples and maximise reconstruction errors in contaminated samples. The experimental results demonstrate that our method outperforms sha…
Uncertainty Awareness on Unsupervised Domain Adaptation for Time Series Data
College of Computing and Data Science, Nanyang Technological University (NTU), Singapore（南洋理工大学计算与数据科学学院）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（江西财经大学计算机与人工智能学院）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息通信研究所（I2R），A*STAR，新加坡）；School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息科学技术大学人工智能学院）
We investigate the use of a non-parametric independence measure, the Hilbert-Schmidt Independence Criterion (HSIC), as a loss-function for learning robust regression and classification models. This loss-function encourages learning models where the distribution of the residuals between the label and the model prediction is statistically independent of the distribution of the instances themselves. This loss-function was first proposed by Mooij et al. (2009) in the context of learning causal graphs. We adapt it to the task of learning for unsupervised covariate shift: learning on a source domain…
We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using…
Calibrated One-class Classification for Unsupervised Time Series Anomaly Detection
National University of Defense Technology（国防科技大学）；College of Computer, National University of Defense Technology（国防科技大学计算机学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Singapore Management University（新加坡管理大学）
The uncertainty estimation is critical in real-world decision making applications, especially when distributional shift between the training and test data are prevalent. Many calibration methods in the literature have been proposed to improve the predictive uncertainty of DNNs which are generally not well-calibrated. However, none of them is specifically designed to work properly under domain shift condition. In this paper, we propose Unsupervised Temperature Scaling (UTS) as a robust calibration method to domain shift. It exploits unlabeled test samples instead of the training one to adjust t…
This paper aims to improve the explainability of Autoencoder's (AE) predictions by proposing two explanation methods based on the mean and epistemic uncertainty of log-likelihood estimate, which naturally arise from the probabilistic formulation of the AE called Bayesian Autoencoders (BAE). To quantitatively evaluate the performance of explanation methods, we test them in sensor network applications, and propose three metrics based on covariate shift of sensors : (1) G-mean of Spearman drift coefficients, (2) G-mean of sensitivity-specificity of explanation ranking and (3) sensor explanation q…
University of Essex（埃塞克斯大学）；Ulster University（阿尔斯特大学）；Swansea University（斯旺西大学）；California State University Fresno（加州州立大学弗雷斯诺分校）
The non-stationary nature of electroencephalography (EEG) signals makes an EEG-based brain-computer interface (BCI) a dynamic system, thus improving its performance is a challenging task. In addition, it is well-known that due to non-stationarity based covariate shifts, the input data distributions of EEG-based BCI systems change during inter- and intra-session transitions, which poses great difficulty for developments of online adaptive data-driven systems. Ensemble learning approaches have been used previously to tackle this challenge. However, passive scheme based implementation leads to po…
Generalized Conformal Predictive Systems Under Distributional Shifts
IDLab Seminar for Statistics（统计研究所研讨会）；Department of Electronics（电子系）；ETH Zurich（苏黎世联邦理工学院）；Information Systems Zurich, Switzerland（苏黎世信息系统，瑞士）；Ghent University（根特大学）
Established methods for unsupervised representation learning such as variational autoencoders produce none or poorly calibrated uncertainty estimates making it difficult to evaluate if learned representations are stable and reliable. In this work, we present a Bayesian autoencoder for unsupervised representation learning, which is trained using a novel variational lower-bound of the autoencoder evidence. This is maximized using Monte Carlo EM with a variational distribution that takes the shape of a Laplace approximation. We develop a new Hessian approximation that scales linearly with data si…
Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition
Center for Management of Information Technologies, Kagoshima University（鹿儿岛大学信息技术管理中心）
UCL（伦敦大学学院）；Meta AI；UC Berkeley（加州大学伯克利分校）；University of Oxford（牛津大学）；MILA（蒙特利尔学习算法研究所）；Inflection AI（Inflection人工智能）；Cohere；FLAIR（未来人工智能研究中心）
Adaptive curricula in reinforcement learning (RL) have proven effective for producing policies robust to discrepancies between the train and test environment. Recently, the Unsupervised Environment Design (UED) framework generalized RL curricula to generating sequences of entire environments, leading to new methods with robust minimax regret properties. Problematically, in partially-observable or stochastic settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment in the intended deployment setting, while curriculum learning necessarily…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；George Washington University（乔治华盛顿大学）
Recent development in the data-driven decision science has seen great advances in individualized decision making. Given data with individual covariates, treatment assignments and outcomes, policy makers best individualized treatment rule (ITR) that maximizes the expected outcome, known as the value function. Many existing methods assume that the training and testing distributions are the same. However, the estimated optimal ITR may have poor generalizability when the training and testing distributions are not identical. In this paper, we consider the problem of finding an optimal ITR from a re…
Towards regularized learning from functional data with covariate shift
Institute for Machine Learning（机器学习研究所）；Johannes Kepler University（约翰·凯撒大学）；Medical University of Innsbruck（因斯布鲁克医学院）；Austrian Academy of Sciences（奥地利科学院）；Indian Institute of Technology Delhi（德里印度理工学院）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；North Carolina State University（北卡罗来纳州立大学）
Understanding the dependencies among features of a dataset is at the core of most unsupervised learning tasks. However, a majority of generative modeling approaches are focused solely on the joint distribution $p(x)$ and utilize models where it is intractable to obtain the conditional distribution of some arbitrary subset of features $x_u$ given the rest of the observed covariates $x_o$: $p(x_u \mid x_o)$. Traditional conditional approaches provide a model for a fixed set of covariates conditioned on another fixed set of observed covariates. Instead, in this work we develop a model that is cap…
University of Tübingen（蒂宾根大学）；IMPRS-IS（国际马克斯·普朗克智能系统研究学院）；LMU Munich（慕尼黑大学）
Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on th…
A typical desideratum for quantifying the uncertainty from a classification model as a prediction set is class-conditional singleton set calibration. That is, such sets should map to the output of well-calibrated selective classifiers, matching the observed frequencies of similar instances. Recent works proposing adaptive and localized conformal p-values for deep networks do not guarantee this behavior, nor do they achieve it empirically. Instead, we use the strong signals for prediction reliability from KNN-based approximations of Transformer networks to construct data-driven partitions for M…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）
Detecting changes on the ground in multitemporal Earth observation data is one of the key problems in remote sensing. In this paper, we introduce Sibling Regression for Optical Change detection (SiROC), an unsupervised method for change detection in optical satellite images with medium and high resolution. SiROC is a spatial context-based method that models a pixel as a linear combination of its distant neighbors. It uses this model to analyze differences in the pixel and its spatial context-based predictions in subsequent time periods for change detection. We combine this spatial context-base…
Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models
Instituto de Investigación en Ciencias de la Computación, CONICET-UBA（国家科学技术研究委员会-布宜诺斯艾利斯大学计算机科学研究所）；Departamento de Electrónica, Facultad de Ingeniería, Universidad de Buenos Aires（布宜诺斯艾利斯大学工程学院电子系）；Centro de Simulación Computacional para Aplicaciones Tecnológicas, CONICET（国家科学技术研究委员会技术应用计算模拟中心）；Laboratoire des Signaux et Systèmes (L2S), CentraleSupelec CNRS Universite Paris-Saclay（巴黎萨克雷大学中央理工-高等电力学院法国国家科学研究中心信号与系统实验室（L2S））
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）；Le Mans University（勒芒大学）
Pseudo-label (PL) filtering forms a crucial part of Self-Training (ST) methods for unsupervised domain adaptation. Dropout-based Uncertainty-driven Self-Training (DUST) proceeds by first training a teacher model on source domain labeled data. Then, the teacher model is used to provide PLs for the unlabeled target domain data. Finally, we train a student on augmented labeled and pseudo-labeled data. The process is iterative, where the student becomes the teacher for the next DUST iteration. A crucial step that precedes the student model training in each DUST iteration is filtering out noisy PLs…
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning
Shanghai AI Lab（上海人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）
