This paper studies the prediction of a target $\mathbf{z}$ from a pair of random variables $(\mathbf{x},\mathbf{y})$, where the ground-truth predictor is additive $\mathbb{E}[\mathbf{z} \mid \mathbf{x},\mathbf{y}] = f_\star(\mathbf{x}) +g_{\star}(\mathbf{y})$. We study the performance of empirical risk minimization (ERM) over functions $f+g$, $f \in F$ and $g \in G$, fit on a given training distribution, but evaluated on a test distribution which exhibits covariate shift. We show that, when the class $F$ is "simpler" than $G$ (measured, e.g., in terms of its metric entropy), our predictor is more resilient to heterogeneous covariate shifts} in which the shift in $\mathbf{x}$ is much greater than that in $\mathbf{y}$. Our analysis proceeds by demonstrating that ERM behaves qualitatively similarly to orthogonal machine learning: the rate at which ERM recovers the $f$-component of the predictor has only a lower-order dependence on the complexity of the class $G$, adjusted for partial non-indentifiability introduced by the additive structure. These results rely on a novel Hölder style inequality for the Dudley integral which may be of independent interest. Moreover, we corroborate our theoretical findings with experiments demonstrating improved resilience to shifts in "simpler" features across numerous domains.
Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
Department of Biostatistics, Yale University(耶鲁大学生物统计学系);Department of Biostatistics, Harvard University(哈佛大学生物统计学系);Department of Quantitative Methods, Purdue University(普渡大学定量方法系)
Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers
Yau Mathematical Sciences Center, Tsinghua University(清华大学数学科学中心);Khoury College of Computer Sciences, Northeastern University(东北大学计算机科学学院);Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Department of Statistics and Data Science, University of Pennsylvania(宾夕法尼亚大学统计与数据科学系)
Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
Department of Statistics and Operations Research, University of North Carolina, Chapel Hill(统计与运筹学系,北卡罗来纳大学 Chapel Hill 分校);Daniels School of Business, Purdue University(商务学院,普渡大学);Department of Statistics and Operations Research, Department of Genetics, Department of Biostatistics, The University of North Carolina at Chapel Hill(统计与运筹学系,遗传学系,生物统计学系,北卡罗来纳大学 Chapel Hill 分校)
The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging
School of Computation, Information and Technology, Technical University of Munich, Germany(计算、信息与技术学院,慕尼黑技术大学);Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany(生物医学影像机器学习研究所,海德堡慕尼黑研究中心);relAI – Konrad Zuse School of Excellence in Reliable AI(relAI – 卡诺尔·祖斯可靠人工智能卓越学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);School of Biomedical Engineering and Imaging Sciences, King's College London, UK(生物医学工程与成像科学学院,伦敦国王学院);Institute of Pathology, Technical University of Munich, Germany(病理学研究所,慕尼黑技术大学)
Principled Federated Random Forests for Heterogeneous Data
Inria(法国国家信息与自动化研究所);PreMeDICaL(预医学实验室);Inserm(法国国家医学研究院);University of Montpellier(蒙彼利埃大学);Sorbonne Université(索邦大学);Université Paris Cité(巴黎城市大学);CNRS(国家科学研究中心);LPSM(巴黎高等师范学院)
Un-mixing Test-time Adaptation under Heterogeneous Data Streams
Beijing Academy of Artificial Intelligence(北京人工智能研究院);Alibaba Group(阿里巴巴集团);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);Duke Kunshan University(杜克大学昆山分校)
Statistical Learning for Heterogeneous Treatment Effects: Pretraining, Prognosis, and Prediction
Department of Biomedical Data Science, Stanford University(生物医学数据科学系,斯坦福大学);Department of Econometrics & Business Statistics, Monash University(计量经济学与商业统计学系,墨尔本大学);Departments of Biomedical Data Science and of Statistics, Stanford University(生物医学数据科学系和统计学系,斯坦福大学)
Model-Based Reinforcement Learning for Heterogeneous Multi-Robot Task Assignment Under Distribution Shifts
John A. Paulson School Of Engineering And Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院);Harvard Medical School(哈佛医学院);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Stanford University School of Medicine(斯坦福大学医学院)
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Technical University of Munich (TUM)(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);School of Medicine and Health Technical University of Munich (TUM)(医学与健康学院慕尼黑技术大学);Department of Radiation Oncology TUM University Hospital(放射肿瘤科TUM大学医院);Chair for AI in Healthcare and Medicine Technical University of Munich (TUM)(医疗与医学人工智能系慕尼黑技术大学);Department of Pathology School of Medicine Department of Biomedical Engineering School of Engineering University of California Irvine(病理学系医学院生物医学工程系工程学院加州大学伊文斯顿分校)