Robust Generalization despite Distribution Shift via Minimum Discriminating Information

Towards In-distribution Compatibility in Out-of-distribution Detection

Deep neural network, despite its remarkable capability of discriminating targeted in-distribution samples, shows poor performance on detecting anomalous out-of-distribution data. To address this defect, state-of-the-art solutions choose to train deep networks on an auxiliary dataset of outliers. Various training criteria for these auxiliary outliers are proposed based on heuristic intuitions. However, we find that these intuitively designed outlier training criteria can hurt in-distribution learning and eventually lead to inferior performance. To this end, we identify three causes of the in-di…

Minimum weight norm models do not always generalize well for over-parameterized problems

This work is substituted by the paper in arXiv:2011.14066. Stochastic gradient descent is the de facto algorithm for training deep neural networks (DNNs). Despite its popularity, it still requires fine tuning in order to achieve its best performance. This has led to the development of adaptive methods, that claim automatic hyper-parameter optimization. Recently, researchers have studied both algorithmic classes via toy examples: e.g., for over-parameterized linear regression, Wilson et. al. (2017) shows that, while SGD always converges to the minimum-norm solution, adaptive methods show no suc…

Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression

A fundamental principle of learning theory is that there is a trade-off between the complexity of a prediction rule and its ability to generalize. Modern machine learning models do not obey this paradigm: They produce an accurate prediction even with a perfect fit to the training set. We investigate over-parameterized linear regression models focusing on the minimum norm solution: This is the solution with the minimal norm that attains a perfect fit to the training set. We utilize the recently proposed predictive normalized maximum likelihood (pNML) learner which is the min-max regret solution…

Effort-free Automated Skeletal Abnormality Detection of Rat Fetuses on Whole-body Micro-CT Scans

Machine Learning-based fast and quantitative automated screening plays a key role in analyzing human bones on Computed Tomography (CT) scans. However, despite the requirement in drug safety assessment, such research is rare on animal fetus micro-CT scans due to its laborious data collection and annotation. Therefore, we propose various bone feature engineering techniques to thoroughly automate the skeletal localization/labeling/abnormality detection of rat fetuses on whole-body micro-CT scans with minimum effort. Despite limited training data of 49 fetuses, in skeletal labeling and abnormality…

SGD学习可证明在线性可分数据上推广的过参数化网络

SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data

神经网络表现出良好的泛化行为在过参数化制度，其中网络参数的数量超过观察的数量。尽管如此，目前神经网络的泛化界限无法解释这一现象。为了弥补这一差距，我们研究了学习一个两层过参数化神经网络的问题，当数据是由一个线性可分函数。在网络具有Leaky ReLU激活的情况下，我们为过参数化网络提供优化和泛化保证。具体来说，我们证明了SGD的收敛速度的全局最小值，并提供泛化保证这个全局最小值是独立的网络大小。因此，我们的结果清楚地表明，使用SGD进行优化既找到了全局最小值，又避免了过度拟合，尽管模型容量很大。这是第一次从理论上证明了SGD在学习过指定神经网络分类器时可以避免过拟合。