Behavior of linear L2-boosting algorithms in the vanishing learning rate asymptotic

Infinitesimal gradient boosting

We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learning rate tends to zero and the number of gradient trees is rescaled accordingly. For this purpose, we introduce a new class of randomized regression trees bridging totally randomized trees and Extra Trees and using a softmax distribution for binary splitting. Our main result is the convergence of the associated stochastic algorithm and the characterization of the limiting proced…

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers

This paper establishes a precise high-dimensional asymptotic theory for boosting on separable data, taking statistical and computational perspectives. We consider a high-dimensional setting where the number of features (weak learners) $p$ scales with the sample size $n$, in an overparametrized regime. Under a class of statistical models, we provide an exact analysis of the generalization error of boosting when the algorithm interpolates the training data and maximizes the empirical $\ell_1$-margin. Further, we explicitly pin down the relation between the boosting test error and the optimal Bay…

无穷小梯度提升的大样本理论

A large sample theory for infinitesimal gradient boosting

本文研究无穷小梯度提升在大样本极限下的渐近行为，证明其收敛到由总体分布决定的确定性微分方程过程，并分析测试误差下降与长时间性质。

分阶段提升分布回归

Stagewise Boosting Distributional Regression

本文针对分布回归中梯度提升因梯度消失导致次优模型的问题，提出将分阶段回归与梯度提升结合的分阶段提升型算法，并引入相关性过滤正则化及随机近似以处理大数据，通过模拟和闪电计数数据验证性能。

使用无标签数据的样本最优不可知提升

Sample-Optimal Agnostic Boosting with Unlabeled Data

本文提出了一种计算高效的不可知boosting算法，通过引入多项式级数量的额外无标签样本，使其样本复杂度与ERM相匹配，且总样本数不劣于已知最佳算法，解决了现有方法在不可知设置下未达最优样本复杂度的问题。

Online Multiclass Boosting with Bandit Feedback

We present online boosting algorithms for multiclass classification with bandit feedback, where the learner only receives feedback about the correctness of its prediction. We propose an unbiased estimate of the loss using a randomized prediction, allowing the model to update its weak learners with limited information. Using the unbiased estimate, we extend two full information boosting algorithms (Jung et al., 2017) to the bandit setting. We prove that the asymptotic error bounds of the bandit algorithms exactly match their full information counterparts. The cost of restricted feedback is refl…