arXiv math/06034472006-03math.STstat.TH

Optimal rates of aggregation in classification under low noise assumption

Guillaume Lecué

发表信息：: Bernoulli 13, 4 (2007) 1000-1022
DOI：: 10.3150/07-BEJ6044

英文摘要：

In the same spirit as Tsybakov (2003), we define the optimality of an aggregation procedure in the problem of classification. Using an aggregate with exponential weights, we obtain an optimal rate of convex aggregation for the hinge risk under the margin assumption. Moreover we obtain an optimal rate of model selection aggregation under the margin assumption for the excess Bayes risk.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Optimal oracle inequality for aggregation of classifiers under low noise condition

We consider the problem of optimality, in a minimax sense, and adaptivity to the margin and to regularity in binary classification. We prove an oracle inequality, under the margin assumption (low noise condition), satisfied by an aggregation procedure which uses exponential weights. This oracle inequality has an optimal residual: $(\log M/n)^{κ/(2κ-1)}$ where $κ$ is the margin parameter, $M$ the number of classifiers to aggregate and $n$ the number of observations. We use this inequality first to construct minimax classifiers under margin and regularity assumptions and second to aggregate them…

2006-03-22math.STstat.TH

PAC-Bayesian aggregation of affine estimators

Aggregating estimators using exponential weights depending on their risk appears optimal in expectation but not in probability. We use here a slight overpenalization to obtain oracle inequality in probability for such an explicit aggregation procedure. We focus on the fixed design regression framework and the aggregation of affine estimators and obtain results for a large family of affine estimators under a non necessarily independent sub-Gaussian noise assumptions.

2018-01-31math.STstat.TH

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

We study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp PAC-Bayesian risk bounds for aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We then apply these results to derive sparsity oracle inequalities.

2013-03-22math.STstat.TH

Optimal Kullback-Leibler Aggregation in Mixture Density Estimation by Maximum Likelihood

We study the maximum likelihood estimator of density of $n$ independent observations, under the assumption that it is well approximated by a mixture with a large number of components. The main focus is on statistical properties with respect to the Kullback-Leibler loss. We establish risk bounds taking the form of sharp oracle inequalities both in deviation and in expectation. A simple consequence of these bounds is that the maximum likelihood estimator attains the optimal rate $((\log K)/n)^{1/2}$, up to a possible logarithmic correction, in the problem of convex aggregation when the number $K…

2017-01-18math.STstat.TH

Contemporaneous aggregation of triangular array of random-coefficient AR(1) processes

We discuss contemporaneous aggregation of independent copies of a triangular array of random-coefficient AR(1) processes with i.i.d. innovations belonging to the domain of attraction of an infinitely divisible law W. The limiting aggregated process is shown to exist, under general assumptions on W and the mixing distribution, and is represented as a mixed infinitely divisible moving-average. Partial sums process of $ is discussed under the assumption E(W^2) is finite and a mixing density regularly varying at the "unit root" x=1 with exponent β>0. We show that the above partial sums process may…

2013-07-06math.STstat.TH

On optimality of empirical risk minimization in linear aggregation

In the first part of this paper, we show that the small-ball condition, recently introduced by Mendelson (2015), may behave poorly for important classes of localized functions such as wavelets, piecewise polynomials or trigonometric polynomials, in particular leading to suboptimal estimates of the rate of convergence of ERM for the linear aggregation problem. In a second part, we recover optimal rates of covergence for the excess risk of ERM when the dictionary is made of trigonometric functions. Considering the bounded case, we derive the concentration of the excess risk around a single point…

2016-10-18math.STstat.TH

更多相关论文收起更多论文

Aggregation of penalized empirical risk minimizers in regression

We give a general result concerning the rates of convergence of penalized empirical risk minimizers (PERM) in the regression model. Then, we consider the problem of agnostic learning of the regression, and give in this context an oracle inequality and a lower bound for PERM over a finite class. These results hold for a general multivariate random design, the only assumption being the compactness of the support of its law (allowing discrete distributions for instance). Then, using these results, we construct adaptive estimators. We consider as examples adaptive estimation over anisotropic Besov…

2008-10-29math.STstat.TH

On aggregation for heavy-tailed classes

We introduce an alternative to the notion of `fast rate' in Learning Theory, which coincides with the optimal error rate when the given class happens to be convex and regular in some sense. While it is well known that such a rate cannot always be attained by a learning procedure (i.e., a procedure that selects a function in the given class), we introduce an aggregation procedure that attains that rate under rather minimal assumptions -- for example, that the $L_q$ and $L_2$ norms are equivalent on the linear span of the class for some $q>2$, and the target random variable is square-integrable.

2015-02-25math.STstat.MLstat.TH

Simple proof of the risk bound for denoising by exponential weights for asymmetric noise distributions

In this note, we consider the problem of aggregation of estimators in order to denoise a signal. The main contribution is a short proof of the fact that the exponentially weighted aggregate satisfies a sharp oracle inequality. While this result was already known for a wide class of symmetric noise distributions, the extension to asymmetric distributions presented in this note is new.

2022-12-25math.STstat.TH

Optimal exponential bounds for aggregation of density estimators

We consider the problem of model selection type aggregation in the context of density estimation. We first show that empirical risk minimization is sub-optimal for this problem and it shares this property with the exponential weights aggregate, empirical risk minimization over the convex hull of the dictionary functions, and all selectors. Using a penalty inspired by recent works on the $Q$-aggregation procedure, we derive a sharp oracle inequality in deviation under a simple boundedness assumption and we show that the rate is optimal in a minimax sense. Unlike the procedures based on exponent…

2016-09-28math.STstat.TH

Sparse Regression Learning by Aggregation and Langevin Monte-Carlo

We consider the problem of regression learning for deterministic design and independent random errors. We start by proving a sharp PAC-Bayesian type bound for the exponentially weighted aggregate (EWA) under the expected squared empirical loss. For a broad class of noise distributions the presented bound is valid whenever the temperature parameter $β$ of the EWA is larger than or equal to $4σ^2$, where $σ^2$ is the noise variance. A remarkable feature of this result is that it is valid even for unbounded regression functions and the choice of the temperature parameter depends exclusively on th…

2010-02-16math.STstat.APstat.TH

Suboptimality of Penalized Empirical Risk Minimization in Classification

Let $\cF$ be a set of $M$ classification procedures with values in $[-1,1]$. Given a loss function, we want to construct a procedure which mimics at the best possible rate the best procedure in $\cF$. This fastest rate is called optimal rate of aggregation. Considering a continuous scale of loss functions with various types of convexity, we prove that optimal rates of aggregation can be either $((\log M)/n)^{1/2}$ or $(\log M)/n$. We prove that, if all the $M$ classifiers are binary, the (penalized) Empirical Risk Minimization procedures are suboptimal (even under the margin/low noise conditio…

2007-03-27math.STq-fin.RMstat.TH

Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processes

In this work, we study the problem of aggregating a finite number of predictors for nonstationary sub-linear processes. We provide oracle inequalities relying essentially on three ingredients: (1) a uniform bound of the $\ell^1$ norm of the time varying sub-linear coefficients, (2) a Lipschitz assumption on the predictors and (3) moment conditions on the noise appearing in the linear representation. Two kinds of aggregations are considered giving rise to different moment conditions on the noise and more or less sharp oracle inequalities. We apply this approach for deriving an adaptive predicto…

2015-11-17math.STstat.MLstat.TH

统计聚合的局部风险界

Local Risk Bounds for Statistical Aggregation

CREST, ENSAE, Institut Polytechnique de Paris（巴黎理工学院ENSAE Crest实验室）；EPFL（洛桑联邦理工学院）；University of California Berkeley（加州大学伯克利分校）

本文通过用局部复杂度替代全局复杂度，收紧了统计聚合理论中的经典风险界，并基于PAC-Bayes局部化技术改进了指数权重和Q-aggregation估计器的偏差最优界。

2023-06-29cs.ITcs.LGmath.IT