具有平方铰链的完全校正梯度提升：快速学习率和早期停止
Fully-Corrective Gradient Boosting with Squared Hinge: Fast Learning Rates and Early Stopping
摘要：
Boosting是一种众所周知的用于提高机器学习中弱学习器的准确性的方法。然而，它的理论推广保证在文献中缺失。在本文中，我们提出了一个有效的提升方法与理论推广保证二进制分类。所提出的增强方法的三个关键成分是：a）在增强过程中的\textit{fully-corrective greedy}（FCG）更新，b）作为损失函数的可微分\textit{squared hinge}（也称为\textit{truncated quadratic}）函数，以及c）用于相关联的FCG优化的有效的交替方向乘法（ADMM）算法。所使用的平方铰链损失不仅继承了著名的铰链损失的鲁棒性与离群值的分类，但也带来了一些好处的计算实现和理论论证。在一定的稀疏性假设下，我们推导出所提出的boosting方法的${\cal O}((m/\log m)^{-1/4})$阶的快速学习率，如果施加一定的额外噪声假设，可以进一步改进为${\cal O}((m/\log m)^{-1/2})$，其中$m$是样本集的大小。这两种方法的学习率都是现有Boosting类分类方法推广效果最好的。此外，所提出的方法提供了一个有效的早期停止计划。一系列的玩具模拟和真实数据实验进行了验证发展的理论和证明所提出的方法的有效性。
英文摘要：
Boosting is a well-known method for improving the accuracy of weak learners in machine learning. However, its theoretical generalization guarantee is missing in literature. In this paper, we propose an efficient boosting method with theoretical generalization guarantees for binary classification. Three key ingredients of the proposed boosting method are: a) the \textit{fully-corrective greedy} (FCG) update in the boosting procedure, b) a differentiable \textit{squared hinge} (also called \textit{truncated quadratic}) function as the loss function, and c) an efficient alternating direction method of multipliers (ADMM) algorithm for the associated FCG optimization. The used squared hinge loss not only inherits the robustness of the well-known hinge loss for classification with outliers, but also brings some benefits for computational implementation and theoretical justification. Under some sparseness assumption, we derive a fast learning rate of the order ${\cal O}((m/\log m)^{-1/4})$ for the proposed boosting method, which can be further improved to ${\cal O}((m/\log m)^{-1/2})$ if certain additional noise assumption is imposed, where $m$ is the size of sample set. Both derived learning rates are the best ones among the existing generalization results of boosting-type methods for classification. Moreover, an efficient early stopping scheme is provided for the proposed method. A series of toy simulations and real data experiments are conducted to verify the developed theories and demonstrate the effectiveness of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
