Support Vector Machines (SVMs) are among the most fundamental tools for binary classification. In its simplest formulation, an SVM produces a hyperplane separating two classes of data using the largest possible margin to the data. The focus on maximizing the margin has been well motivated through numerous generalization bounds. In this paper, we revisit and improve the classic generalization bounds in terms of margins. Furthermore, we complement our new generalization bound by a nearly matching lower bound, thus almost settling the generalization performance of SVMs in terms of margins.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）；University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
An information-theoretic upper bound on the generalization error of supervised learning algorithms is derived. The bound is constructed in terms of the mutual information between each individual training sample and the output of the learning algorithm. The bound is derived under more general conditions on the loss function than in existing studies; nevertheless, it provides a tighter characterization of the generalization error. Examples of learning algorithms are provided to demonstrate the the tightness of the bound, and to show that it has a broad range of applicability. Application to nois…
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works. In this paper, we develop a compression approach based on quantizing neural network parameters in a linear subspace, profoundly improving on previous results to provide state-of-the-art generalization bounds on a variety of tasks, including transfer learning. We use these tight bounds to better understand the role of model size, equivariance, and the implicit biases of optimization, for generalization in deep learning. Nota…
For linear classifiers, the relationship between (normalized) output margin and generalization is captured in a clear and simple bound -- a large output margin implies good generalization. Unfortunately, for deep models, this relationship is less clear: existing analyses of the output margin give complicated bounds which sometimes depend exponentially on depth. In this work, we propose to instead analyze a new notion of margin, which we call the "all-layer margin." Our analysis reveals that the all-layer margin has a clear and direct relationship with generalization for deep models. This enabl…
Boosting is one of the most successful ideas in machine learning, achieving great practical performance with little fine-tuning. The success of boosted classifiers is most often attributed to improvements in margins. The focus on margin explanations was pioneered in the seminal work by Schapire et al. (1998) and has culminated in the $k$'th margin generalization bound by Gao and Zhou (2013), which was recently proved to be near-tight for some data distributions (Gronlund et al. 2019). In this work, we first demonstrate that the $k$'th margin bound is inadequate in explaining the performance of…
Hong Kong University of Science and Technology（香港科技大学）
Margin enlargement over training data has been an important strategy since perceptrons in machine learning for the purpose of boosting the robustness of classifiers toward a good generalization ability. Yet Breiman (1999) showed a dilemma that a uniform improvement on margin distribution does NOT necessarily reduces generalization errors. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed spectrally normalized margins, from a novel perspective based on phase transitions of normalized margin distributions in training dynamics. Normalized margin distributi…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；IBM Research AI（IBM人工智能研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Understanding the generalization of deep neural networks is one of the most important tasks in deep learning. Although much progress has been made, theoretical error bounds still often behave disparately from empirical observations. In this work, we develop margin-based generalization bounds, where the margins are normalized with optimal transport costs between independent random subsets sampled from the training distribution. In particular, the optimal transport cost can be interpreted as a generalization of variance which captures the structural properties of the learned feature space. Our b…
On the Current State of Research in Explaining Ensemble Performance Using Margins
Miami University（迈阿密大学）；University of Alabama（阿拉巴马大学）
经验证据表明，集成，如装袋，助推，随机和旋转森林，一般表现更好的泛化误差比个人分类器。为了解释这种性能，Schapire et al.（1998）基于训练数据的裕度开发了集成的泛化误差的上限，从中得出结论，较大的裕度应导致较低的泛化误差，其他条件相同。许多其他研究人员支持这一假设，并提出了更严格的边界上的泛化误差的基础上的利润率或利润率的功能。例如，Shen和Li（2010）提供的证据表明，投票分类器的泛化误差可以通过增加均值和减少边缘的方差来减少。在这篇文章中，我们提出了几种技术和实证检验是否在解释合奏表演的研究现状举行。我们评估所提出的方法，通过实验与真实和模拟数据集。
A major challenge in modern machine learning is theoretically understanding the generalization properties of overparameterized models. Many existing tools rely on uniform convergence (UC), a property that, when it holds, guarantees that the test loss will be close to the training loss, uniformly over a class of candidate models. Nagarajan and Kolter (2019) show that in certain simple linear and neural-network settings, any uniform convergence bound will be vacuous, leaving open the question of how to prove generalization in settings where UC fails. Our main contribution is proving novel genera…
Univ Lyon, UJM-Saint-Etienne, CNRS, Institut d Optique Graduate School, Laboratoire Hubert Curien UMR 5516（里昂大学，圣艾蒂安大学，法国国家科学研究中心，光学研究所研究生院，于贝尔·居里安实验室）；Université Laval（拉瓦尔大学）
In the PAC-Bayesian literature, the C-Bound refers to an insightful relation between the risk of a majority vote classifier (under the zero-one loss) and the first two moments of its margin (i.e., the expected margin and the voters' diversity). Until now, learning algorithms developed in this framework minimize the empirical version of the C-Bound, instead of explicit PAC-Bayesian generalization bounds. In this paper, by directly optimizing PAC-Bayesian guarantees on the C-Bound, we derive self-bounding majority vote learning algorithms. Moreover, our algorithms based on gradient descent are s…
On the Insufficiency of the Large Margins Theory in Explaining the Performance of Ensemble Methods
Miami University（迈阿密大学）；University of Alabama（阿拉巴马大学）
Boosting和其他集成方法通过加权投票将大量弱分类器组合在一起，以产生更强的预测模型。为了解释boosting算法的成功性能，Schapire等人（1998）表明AdaBoost在增加训练数据的边缘方面特别有效。Schapire等人（1998年）还根据训练数据的裕量提出了任何总体的泛化误差的上限，由此得出结论，在其他条件相同的情况下，裕量越大，泛化误差越小（有时称为“大裕量理论”）。更严格的界限已经推导出来，并加强了大利润理论假设。例如，Wang et al.（2011）提出，特定的利润率实例，如均衡利润率，可以更好地总结利润率分布。这些结果导致许多研究人员考虑直接优化的利润，以改善集成泛化误差与混合的结果。我们表明，大利润率理论是不足以解释的投票分类器的性能。我们通过说明如何在保持复杂度固定的同时，提高集成解决方案的裕度分布，而不提高测试集性能来做到这一点。
INRIA - Département d’Informatique de l’Ecole Normale Supérieure, PSL Research University（法国国家信息与自动化研究所 - 巴黎高等师范学院计算机科学系，巴黎文理研究大学）
Max-margin methods for binary classification such as the support vector machine (SVM) have been extended to the structured prediction setting under the name of max-margin Markov networks ($M^3N$), or more generally structural SVMs. Unfortunately, these methods are statistically inconsistent when the relationship between inputs and labels is far from deterministic. We overcome such limitations by defining the learning problem in terms of a "max-min" margin formulation, naming the resulting method max-min margin Markov networks ($M^4N$). We prove consistency and finite sample generalization boun…
UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, the University of Sydney（悉尼大学工程学院计算机科学学院优必选悉尼人工智能中心）；University of Science and Technology of China（中国科学技术大学）
This paper studies the relationship between generalization and privacy preservation in iterative learning algorithms by two sequential steps. We first establish an alignment between generalization and privacy preservation for any learning algorithm. We prove that $(\varepsilon, δ)$-differential privacy implies an on-average generalization bound for multi-database learning algorithms which further leads to a high-probability bound for any learning algorithm. This high-probability bound also implies a PAC-learnable guarantee for differentially private learning algorithms. We then investigate how…
Maximal Margin Distribution Support Vector Regression with coupled Constraints-based Convex Optimization
Shanghai Tenth People's Hospital（上海市第十人民医院）；Tongji University（同济大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Jilin University（吉林大学）；The Ohio State University（俄亥俄州立大学）
University College London（伦敦大学学院）；Massachusetts Institute of Technology（麻省理工学院）
Generalization error bounds are essential to understanding machine learning algorithms. This paper presents novel expected generalization error upper bounds based on the average joint distribution between the output hypothesis and each input training sample. Multiple generalization error upper bounds based on different information measures are provided, including Wasserstein distance, total variation distance, KL divergence, and Jensen-Shannon divergence. Due to the convexity of the information measures, the proposed bounds in terms of Wasserstein distance and total variation distance are show…
University of Georgia（佐治亚大学）；Zhongnan University of Economics and Law（中南财经政法大学）
Large-margin classifiers are popular methods for classification. We derive the asymptotic expression for the generalization error of a family of large-margin classifiers in the limit of both sample size $n$ and dimension $p$ going to $\infty$ with fixed ratio $α=n/p$. This family covers a broad range of commonly used classifiers including support vector machine, distance weighted discrimination, and penalized logistic regression. Our result can be used to establish the phase transition boundary for the separability of two classes. We assume that the data are generated from a single multivariat…
On Cold Posteriors of Probabilistic Neural Networks: Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
School of Mathematics and Statistics（数学与统计学学院）；Guangxi Normal University（广西师范大学）；School of Mathematics（数学学院）；Hohai University（河海大学）；College of Marine Life Sciences（海洋生命科学学院）；Ocean University of China（中国海洋大学）；State Key Laboratory of Mathematical Sciences（数学科学国家重点实验室）；Academy of Mathematics and Systems Science（数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Multi-class SVMs: From Tighter Data-Dependent Generalization Bounds to Novel Algorithms
City University of Hong Kong（香港城市大学）；Microsoft Research（微软研究院）；TU Berlin（柏林工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Humboldt University of Berlin（柏林洪堡大学）
Understanding generalization is crucial to confidently engineer and deploy machine learning models, especially when deployment implies a shift in the data domain. For such domain adaptation problems, we seek generalization bounds which are tractably computable and tight. If these desiderata can be reached, the bounds can serve as guarantees for adequate performance in deployment. However, in applications where deep neural networks are the models of choice, deriving results which fulfill these remains an unresolved challenge; most existing bounds are either vacuous or has non-estimable terms, e…
In the problem of domain generalization (DG), there are labeled training data sets from several related prediction problems, and the goal is to make accurate predictions on future unlabeled data sets that are not known to the learner. This problem arises in several applications where data distributions fluctuate because of environmental, technical, or other sources of variation. We introduce a formal framework for DG, and argue that it can be viewed as a kind of supervised learning problem by augmenting the original feature space with the marginal distribution of feature vectors. While our fra…
Google Research（谷歌研究院）；Courant Institute of Mathematical Sciences（柯朗数学科学研究所）
We present a series of new and more favorable margin-based learning guarantees that depend on the empirical margin loss of a predictor. We give two types of learning bounds, both distribution-dependent and valid for general families, in terms of the Rademacher complexity or the empirical $\ell_\infty$ covering number of the hypothesis set used. Furthermore, using our relative deviation margin bounds, we derive distribution-dependent generalization bounds for unbounded loss functions under the assumption of a finite moment. We also briefly highlight several applications of these bounds and disc…
We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a $p$-dimensional space given $n$ i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of ${\sqrt{p}}/{\sqrt{n}}$ from prior work and obtain a sharper rate of $\sqrt[4]{p}/\sqrt{n}$. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential priva…
Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）
A learning algorithm referred to as Maximum Margin (MM) is proposed for considering the class-imbalance data learning issue: the trained model tends to predict the majority of classes rather than the minority ones. That is, underfitting for minority classes seems to be one of the challenges of generalization. For a good generalization of the minority classes, we design a new Maximum Margin (MM) loss function, motivated by minimizing a margin-based generalization bound through the shifting decision bound. The theoretically-principled label-distribution-aware margin (LDAM) loss was successfully …
Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；Ben-Gurion University（本-古里安大学）
We analyze a family of supervised learning algorithms based on sample compression schemes that are stable, in the sense that removing points from the training set which were not selected for the compression set does not alter the resulting classifier. We use this technique to derive a variety of novel or improved data-dependent generalization bounds for several learning algorithms. In particular, we prove a new margin bound for SVM, removing a log factor. The new bound is provably optimal. This resolves a long-standing open question about the PAC margin bounds achievable by SVM.
Leveraging algorithmic stability to derive sharp generalization bounds is a classic and powerful approach in learning theory. Since Vapnik and Chervonenkis [1974] first formalized the idea for analyzing SVMs, it has been utilized to study many fundamental learning algorithms (e.g., $k$-nearest neighbors [Rogers and Wagner, 1978], stochastic gradient method [Hardt et al., 2016], linear regression [Maurer, 2017], etc). In a recent line of great works by Feldman and Vondrak [2018, 2019] as well as Bousquet et al. [2020b], they prove a high probability generalization upper bound of order $\tilde{\…
Center for Brains, Minds + Machines（大脑、心智与机器中心）；MIT（麻省理工学院）；Brown University（布朗大学）
Recent theoretical results show that gradient descent on deep neural networks under exponential loss functions locally maximizes classification margin, which is equivalent to minimizing the norm of the weight matrices under margin constraints. This property of the solution however does not fully characterize the generalization performance. We motivate theoretically and show empirically that the area under the curve of the margin distribution on the training set is in fact a good measure of generalization. We then show that, after data separation is achieved, it is possible to dynamically reduc…
