Transfer learning (TL) enables the transfer of knowledge gained in learning to perform one task (source) to a related but different task (target), hence addressing the expense of data acquisition and labeling, potential computational power limitations, and dataset distribution mismatches. We propose a new TL framework for task-specific learning (functional regression in partial differential equations (PDEs)) under conditional shift based on the deep operator network (DeepONet). Task-specific operator learning is accomplished by fine-tuning task-specific layers of the target DeepONet using a hybrid loss function that allows for the matching of individual target samples while also preserving the global properties of the conditional distribution of target data. Inspired by the conditional embedding operator theory, we minimize the statistical distance between labeled target data and the surrogate prediction on unlabeled target data by embedding conditional distributions onto a reproducing kernel Hilbert space. We demonstrate the advantages of our approach for various TL scenarios involving nonlinear PDEs under diverse conditions due to shift in the geometric domain and model dynamics. Our TL framework enables fast and efficient learning of heterogeneous tasks despite significant differences between the source and target domains.
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Domain generalization (DG) aims to learn a model on several source domains, hoping that the model can generalize well to unseen target domains. The distribution shift between domains contains the covariate shift and conditional shift, both of which the model must be able to handle for better generalizability. In this paper, a novel DG method is proposed to deal with the distribution shift via Visual Alignment and Uncertainty-guided belief Ensemble (VAUE). Specifically, for the covariate shift, a visual alignment module is designed to align the distribution of image style to a common empirical …
CNRS-ISIR, Sorbonne University（索邦大学法国国家科学研究中心智能系统与机器人研究所）；Criteo AI Lab（科韬人工智能实验室）；Université de Rouen, LITIS（鲁昂大学LITIS实验室）
We consider the problem of unsupervised domain adaptation (UDA) between a source and a target domain under conditional and label shift a.k.a Generalized Target Shift (GeTarS). Unlike simpler UDA settings, few works have addressed this challenging problem. Recent approaches learn domain-invariant representations, yet they have practical limitations and rely on strong assumptions that may not hold in practice. In this paper, we explore a novel and general approach to align pretrained representations, which circumvents existing drawbacks. Instead of constraining representation invariance, it lear…
Warsaw University of Technology（华沙理工大学）；Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究所（ISTA））
We introduce LIMES, a new method for learning with non-stationary streaming data, inspired by the recent success of meta-learning. The main idea is not to attempt to learn a single classifier that would have to work well across all occurring data distributions, nor many separate classifiers, but to exploit a hybrid strategy: we learn a single set of model parameters from which a specific classifier for any specific data distribution is derived via classifier adaptation. Assuming a multi-class classification setting with class-prior shift, the adaptation step can be performed analytically with …
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；National University of Singapore（新加坡国立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this work, we propose a domain generalization (DG) approach to learn on several labeled source domains and transfer knowledge to a target domain that is inaccessible in training. Considering the inherent conditional and label shifts, we would expect the alignment of $p(x|y)$ and $p(y)$. However, the widely used domain invariant feature learning (IFL) methods relies on aligning the marginal concept shift w.r.t. $p(x)$, which rests on an unrealistic assumption that $p(y)$ is invariant across domains. We thereby propose a novel variational Bayesian inference framework to enforce the conditiona…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Alibaba Group（阿里巴巴集团）；Carnegie Mellon University（卡内基梅隆大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Southern California（南加州大学）
In this work, we propose an adversarial unsupervised domain adaptation (UDA) approach with the inherent conditional and label shifts, in which we aim to align the distributions w.r.t. both $p(x|y)$ and $p(y)$. Since the label is inaccessible in the target domain, the conventional adversarial UDA assumes $p(y)$ is invariant across domains, and relies on aligning $p(x)$ as an alternative to the $p(x|y)$ alignment. To address this, we provide a thorough theoretical and empirical analysis of the conventional adversarial UDA methods under both conditional and label shifts, and propose a novel and p…
As a vital problem in classification-oriented transfer, unsupervised domain adaptation (UDA) has attracted widespread attention in recent years. Previous UDA methods assume the marginal distributions of different domains are shifted while ignoring the discriminant information in the label distributions. This leads to classification performance degeneration in real applications. In this work, we focus on the conditional distribution shift problem which is of great concern to current conditional invariant models. We aim to seek a kernel covariance embedding for conditional distribution which rem…
Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior
Department of Statistics and Data Science, Fudan University（复旦大学统计与数据科学系）；Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania（宾夕法尼亚大学生物统计学、流行病学与信息学系）；Department of Statistics and Applied Probability, University of California, Santa Barbara（加州大学圣芭芭拉分校统计学与应用概率系）
Microsoft Research Montreal（微软研究院蒙特利尔）；D. E. Shaw & Co.（D. E. Shaw公司）；UC Santa Barbara（加州大学圣巴巴拉分校）
Adversarial learning has demonstrated good performance in the unsupervised domain adaptation setting, by learning domain-invariant representations. However, recent work has shown limitations of this approach when label distributions differ between the source and target domains. In this paper, we propose a new assumption, generalized label shift ($GLS$), to improve robustness against mismatched label distributions. $GLS$ states that, conditioned on the label, there exists a representation of the input that is invariant between the source and target domains. Under $GLS$, we provide theoretical g…
University of Chinese Academy of Sciences（中国科学院大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Recently, generalization on out-of-distribution (OOD) data with correlation shift has attracted great attentions. The correlation shift is caused by the spurious attributes that correlate to the class label, as the correlation between them may vary in training and test data. For such a problem, we show that given the class label, the models that are conditionally independent of spurious attributes are OOD generalizable. Based on this, a metric Conditional Spurious Variation (CSV) which controls the OOD generalization error, is proposed to measure such conditional independence. To improve the O…
A generalized approach to label shift: the Conditional Probability Shift Model
Institute of Computer Science, Polish Academy of Sciences（波兰科学院计算机科学研究所）；Faculty of Mathematics and Information Science, Warsaw University of Technology（华沙理工大学数学与信息科学学院）
While previous distribution shift detection approaches can identify if a shift has occurred, these approaches cannot localize which specific features have caused a distribution shift -- a critical step in diagnosing or fixing any underlying issue. For example, in military sensor networks, users will want to detect when one or more of the sensors has been compromised, and critically, they will want to know which specific sensors might be compromised. Thus, we first define a formalization of this problem as multiple conditional distribution hypothesis tests and propose both non-parametric and pa…
Criteo AI Lab（Criteo人工智能实验室）；Ecole Polytechnique（巴黎综合理工学院）；INSA de Rouen Normandie（鲁昂国立应用科学学院）；Université Technologique de Compiègne（贡比涅技术大学）；Université de Rouen Normandie（鲁昂诺曼底大学）；Université de Bretagne Sud（南布列塔尼大学）
We address the problem of unsupervised domain adaptation under the setting of generalized target shift (joint class-conditional and label shifts). For this framework, we theoretically show that, for good generalization, it is necessary to learn a latent representation in which both marginals and class-conditional distributions are aligned across domains. For this sake, we propose a learning problem that minimizes importance weighted loss in the source domain and a Wasserstein distance between weighted marginals. For a proper weighting, we provide an estimator of target label proportion by blen…
As a fundamental problem in machine learning, dataset shift induces a paradigm to learn and transfer knowledge under changing environment. Previous methods assume the changes are induced by covariate, which is less practical for complex real-world data. We consider the Generalized Label Shift (GLS), which provides an interpretable insight into the learning and transfer of desirable knowledge. Current GLS methods: 1) are not well-connected with the statistical learning theory; 2) usually assume the shifting conditional distributions will be matched with an implicit transformation, but its expli…
Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network
Department of Mechanical Engineering, University of Wisconsin–Madison（机械工程系，威斯康星大学麦迪逊分校）；Department of Mathematics, University of Wisconsin–Madison（数学系，威斯康星大学麦迪逊分校）
SPDIM: Source-Free Unsupervised Conditional and Label Shift Adaptation in EEG
Nara Institute of Science and Technology (NAIST)（奈良科学技术大学）；Department of Dynamic Brain Imaging, ATR（动态脑成像部门，ATR）；Center for Advanced Intelligence Project, RIKEN（先进智能项目中心，RIKEN）
Score-based Conditional Out-of-Distribution Augmentation for Graph Covariate Shift
Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗克琴堡研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒姆研究实验室）；The Pennsylvania State University（宾夕法尼亚州立大学）
When training a deep neural network for image classification, one can broadly distinguish between two types of latent features of images that will drive the classification. We can divide latent features into (i) "core" or "conditionally invariant" features $X^\text{core}$ whose distribution $X^\text{core}\vert Y$, conditional on the class $Y$, does not change substantially across domains and (ii) "style" features $X^{\text{style}}$ whose distribution $X^{\text{style}} \vert Y$ can change substantially across domains. Examples for style features include position, rotation, image quality or brig…
Audited Conformal Prediction for Classification under Unknown Distribution Shift
Department of Data Sciences and Operations, University of Southern California（数据科学与运营系，南加州大学）；AI Foundations, Capital One（Capital One人工智能基础）；Department of Data Sciences and Operations, Thomas Lord Department of Computer Science, University of Southern California（数据科学与运营系，托马斯·劳德计算机科学系，南加州大学）
Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition
Center for Management of Information Technologies, Kagoshima University（鹿儿岛大学信息技术管理中心）
Parameter-Free and Group Conditional Online Conformal Prediction
Data Science and AI Institute (DSAI), Johns Hopkins University（数据科学与人工智能研究院（DSAI），约翰霍普金斯大学）；Mathematical Institute for Data Science (MINDS), Johns Hopkins University（数据科学数学研究院（MINDS），约翰霍普金斯大学）；Department of Biomedical Engineering, Johns Hopkins University（生物医学工程系，约翰霍普金斯大学）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Amazon Responsible AI（亚马逊负责任人工智能）
Multi-CATE: Multi-Accurate Conditional Average Treatment Effect Estimation Robust to Unknown Covariate Shifts
Ludwig-Maximilians-University of Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Cornell University（康奈尔大学）；University of Southern California（南加州大学）
Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
We consider the filtering and prediction problem for a diffusion process. The signal and observation are modeled by stochastic differential equations (SDEs) driven by correlated Wiener processes. In classical estimation theory, measure-valued stochastic partial differential equations (SPDEs) are derived for the filtering and prediction measures. These equations can be hard to solve numerically. We provide an approximation algorithm using conditional generative adversarial networks (GANs) in combination with signatures, an object from rough path theory. The signature of a sufficiently smooth pa…
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tencent AI Lab（腾讯人工智能实验室）
This work studies black-box adversarial attacks against deep neural networks (DNNs), where the attacker can only access the query feedback returned by the attacked DNN model, while other information such as model parameters or the training datasets are unknown. One promising approach to improve attack performance is utilizing the adversarial transferability between some white-box surrogate models and the target model (i.e., the attacked model). However, due to the possible differences on model architectures and training datasets between surrogate and target models, dubbed "surrogate biases", t…
Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations
University of Stuttgart（斯图加特大学）；Stuttgart Center for Simulation Science (SimTech)（斯图加特模拟科学中心）；International Max Planck Research School for Intelligent Systems (IMPRS-IS)（国际马克斯·普朗克智能系统研究院）
