Mixture Data for Training Cannot Ensure Out-of-distribution Generalization
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Chongqing Jiaotong University(重庆交通大学);Hong Kong Baptist University(香港浸会大学);National University of Defense Technology(国防科技大学);The University of Sydney(悉尼大学)
Songming Zhang, Yuxiao Luo, Qizhou Wang, Haoang Chi, Xiaofeng Chen, Bo Han, Jinyan Li
Deep neural networks often face generalization problems to handle out-of-distribution (OOD) data, and there remains a notable theoretical gap between the contributing factors and their respective impacts. Literature evidence from in-distribution data has suggested that generalization error can shrink if the size of mixture data for training increases. However, when it comes to OOD samples, this conventional understanding does not hold anymore -- Increasing the size of training data does not always lead to a reduction in the test generalization error. In fact, diverse trends of the errors have been found across various shifting scenarios including those decreasing trends under a power-law pattern, initial declines followed by increases, or continuous stable patterns. Previous work has approached OOD data qualitatively, treating them merely as samples unseen during training, which are hard to explain the complicated non-monotonic trends. In this work, we quantitatively redefine OOD data as those situated outside the convex hull of mixed training data and establish novel generalization error bounds to comprehend the counterintuitive observations better. Our proof of the new risk bound agrees that the efficacy of well-trained models can be guaranteed for unseen data within the convex hull; More interestingly, but for OOD data beyond this coverage, the generalization cannot be ensured, which aligns with our observations. Furthermore, we attempted various OOD techniques to underscore that our results not only explain insightful observations in recent OOD generalization work, such as the significance of diverse data and the sensitivity to unseen shifts of existing algorithms, but it also inspires a novel and effective data selection strategy.
Mixture of Decision Trees for Interpretable Machine Learning
University of Münster(明斯特大学);Fraunhofer Institute for Manufacturing Engineering and Automation IPA(弗劳恩霍夫制造工程与自动化研究所IPA);Dresden University of Technology(德累斯顿工业大学);University of Stuttgart(斯图加特大学)
Score-of-Mixture Training: Training One-Step Generative Models Made Simple via Score Estimation of Mixture Distributions
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, USA(电气工程与计算机科学系,麻省理工学院(MIT),剑桥,马萨诸塞州,美国)
提出Score-of-Mixture Training (SMT)框架,通过最小化$α$-skew Jensen-Shannon散度并估计混合分布得分来训练单步生成模型,支持从零开始训练与预训练扩散模型蒸馏,实验证明其在CIFAR-10和ImageNet 64x64上表现优异。
Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference
School of Mathematical Sciences, Peking University, Beijing, China(北京大学数学科学学院);Department of Mathematics, University of California, Los Angeles, CA, USA(加州大学洛杉矶分校数学系);Beijing International Center for Mathematical Research, Center for Machine Learning Research, Peking University, Beijing, China(北京国际数学研究中心,机器学习研究中心,北京大学,北京,中国)
Variational Learning for the Inverted Beta-Liouville Mixture Model and Its Application to Text Categorization
Hezhou University(贺州学院);North China University of Technology(北方工业大学);DigApis Information Security Technology Co., Ltd.(江苏迪格艾普斯信息安全技术有限公司);Jiangsu University(江苏大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Bayesian Jammer Localization with a Hybrid CNN and Path-Loss Mixture of Experts
1Dept. of Electrical \& Computer Engineering, Northeastern University, Boston, MA, USA 2Dept. of Computer Science, University of Massachusetts Boston, Boston, MA, USA
New York University Abu Dhabi(纽约大学阿布扎比分校);New York University Tandon School of Engineering(纽约大学坦登工程学院);University of Science and Technology of China(中国科学技术大学);BITS Pilani(博拉理工学院皮拉尼校区)
Theoretical Error Analysis of Entropy Approximation for Gaussian Mixtures
Education and Research Center for Mathematical and Data Science, Shimane University(岛根大学数学与数据科学教育研究中心);Graduate School of Mathematics, Nagoya University(名古屋大学数学研究生院);Mathematical and Systems Engineering, Shizuoka University(静冈大学数学与系统工程系);Graduate School of Biomedical Engineering, Tohoku University(东北大学生物医学工程研究生院);AISIN SOFTWARE(爱信软件公司)
Institut Camille Jordan(卡米尔·若尔当研究所);École Centrale de Lyon(里昂中央理工学院);Toulouse School of Economics(图卢兹经济学院);Institut Universitaire de France(法兰西大学研究院);CNRS(法国国家科学研究中心);INSA(法国国立应用科学学院)