This paper investigates asymptotic properties of algorithms that can be viewed as robust analogues of the classical empirical risk minimization. These strategies are based on replacing the usual empirical average by a robust proxy of the mean, such as the (version of) the median of means estimator. It is well known by now that the excess risk of resulting estimators often converges to zero at optimal rates under much weaker assumptions than those required by their ``classical'' counterparts. However, less is known about the asymptotic properties of the estimators themselves, for instance, whether robust analogues of the maximum likelihood estimators are asymptotically efficient. We make a step towards answering these questions and show that for a wide class of parametric problems, minimizers of the appropriately defined robust proxy of the risk converge to the minimizers of the true risk at the same rate, and often have the same asymptotic variance, as the estimators obtained by minimizing the usual empirical risk.
Operational risk models commonly employ maximum likelihood estimation (MLE) to fit loss data to heavy-tailed distributions. Yet several desirable properties of MLE (e.g. asymptotic normality) are generally valid only for large sample-sizes, a situation rarely encountered in operational risk. In this paper, we study how asymptotic normality does--or does not--hold for common severity distributions in operational risk models. We then apply these results to evaluate errors caused by failure of asymptotic normality in constructing confidence intervals around the MLE fitted parameters.
University of Copenhagen（哥本哈根大学）；Delft University of Technology（代尔夫特理工大学）
Plotting a learner's average performance against the number of training samples results in a learning curve. Studying such curves on one or more data sets is a way to get to a better understanding of the generalization properties of this learner. The behavior of learning curves is, however, not very well understood and can display (for most researchers) quite unexpected behavior. Our work introduces the formal notion of \emph{risk monotonicity}, which asks the risk to not deteriorate with increasing training set sizes in expectation over the training samples. We then present the surprising res…
Entropy integrals are widely used as a powerful empirical process tool to obtain upper bounds for the rates of convergence of global empirical risk minimizers (ERMs), in standard settings such as density estimation and regression. The upper bound for the convergence rates thus obtained typically matches the minimax lower bound when the entropy integral converges, but admits a strict gap compared to the lower bound when it diverges. Birgé and Massart [BM93] provided a striking example showing that such a gap is real with the entropy structure alone: for a variant of the natural Hölder class wit…
We use the so-called t-Hill tail index estimator proposed by Fabián(2001), rather than Hill's one, to derive a robust estimator for the distortion risk premium of loss. Under the second-order condition of regular variation, we establish its asymptotic normality. By simulation study, we show that this new estimator is more robust than of Necir and Meraghni 2009 both for small and large samples.
One of the most commonly used methods for forming confidence intervals for statistical inference is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator is unknown. However, despite its ubiquitous role, its theoretical properties are still not well understood for non-asymptotically normal estimators. In this paper, under stability conditions, we establish the limiting distribution of the empirical bootstrap estimator, derive tight conditions for it to be asymptotically consistent, and quantify the speed of convergence. Moreover, we propose thre…
Expectiles were introduced by Newey and Powell (1987) in the context of linear regression models. Recently, Bellini et al. (2014) revealed that expectiles can also be seen as reasonable law-invariant risk measures. In this article, we show that the corresponding statistical functionals are continuous w.r.t. the $1$-weak topology and suitably functionally differentiable. By means of these regularity results we can derive several properties as consistency, asymptotic normality, bootstrap consistency, and qualitative robustness of the corresponding estimators in nonparametric and parametric stati…
In proofs of L_2-differentiability, Lebesgue densities of a central distribution are often assumed right from the beginning. Generalizing Theorem 4.2 of Huber[81], we show that in the class of smooth parametric group models these densities are in fact consequences of a finite Fisher information of the model, provided a suitable representation of the latter is used. The proof uses the notions of absolute continuity in k dimensions and weak differentiability. As examples to which this theorem applies, we spell out a number of models including a correlation model and the general multivariate loca…
We consider estimation of a functional of the data distribution based on i.i.d. observations. We assume the target function can be defined as the minimizer of the expectation of a loss function over a class of $d$-variate real valued cadlag functions that have finite sectional variation norm. For all $k=0,1,\ldots$, we define a $k$-th order smoothness class of functions as $d$-variate functions on the unit cube for which each of a sequentially defined $k$-th order Radon-Nikodym derivative w.r.t. Lebesgue measure is cadlag and of bounded variation. For a target function in this $k$-th order smo…
The asymptotic solution to the problem of comparing the means of two heteroscedastic populations, based on two random samples from the populations, hinges on the pivot underpinning the construction of the confidence interval and the test statistic being asymptotically standard Normal, which is known to happen if the two samples are independent and the ratio of the sample sizes converges to a finite positive number. This restriction on the asymptotic behavior of the ratio of the sample sizes carries the risk of rendering the asymptotic justification of the finite sample approximation invalid. I…
We study Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) for regression problems with convex and $L$-Lipschitz loss functions. We consider a setting where $|\cO|$ malicious outliers contaminate the labels. In that case, under a local Bernstein condition, we show that the $L_2$-error rate is bounded by $ r_N + AL |\cO|/N$, where $N$ is the total number of observations, $r_N$ is the $L_2$-error rate in the non-contaminated setting and $A$ is a parameter coming from the local Bernstein condition. When $r_N$ is minimax-rate-optimal in a non-contaminated setting, th…
We prove the strong consistency and the asymptotic normality of the maximum likelihood estimator of the parameters of a general conditionally heteroscedastic model with $α$-stable innovations. Then, we relax the assumptions and only suppose that the innovation process converges in distribution toward a stable process. Using a pseudo maximum likelihood estimator with a stable density, we also obtain the strong consistency and the asymptotic normality of the estimator. This framework seems relevant for financial data exhibiting heavy tails. We apply this method to several financial index and com…
The aim of this paper is to generalize the PAC-Bayesian theorems proved by Catoni in the classification setting to more general problems of statistical inference. We show how to control the deviations of the risk of randomized estimators. A particular attention is paid to randomized estimators drawn in a small neighborhood of classical estimators, whose study leads to control the risk of the latter. These results allow to bound the risk of very general estimation procedures, as well as to perform model selection.
The effect of measurement errors in discriminant analysis is investigated. Given observations $Z=X+ε$, where $ε$ denotes a random noise, the goal is to predict the density of $X$ among two possible candidates $f$ and $g$. We suppose that we have at our disposal two learning samples. The aim is to approach the best possible decision rule $G^\star$ defined as a minimizer of the Bayes risk. In the free-noise case $(ε=0)$, minimax fast rates of convergence are well-known under the margin assumption in discriminant analysis (see \cite{mammen}) or in the more general classification framework (see \c…
We establish a new concentration result for regularized risk minimizers which is similar to an oracle inequality. Applying this inequality to regularized least squares minimizers like least squares support vector machines, we show that these algorithms learn with (almost) the optimal rate in some specific situations. In addition, for regression our results suggest that using the loss function $L_α(y,t)=|y-t|^α$ with $α$ near 1 may often be preferable to the usual choice of $α=2$.
Various members of the class of weighted insurance premiums and risk capital allocation rules have been researched from a number of perspectives. Corresponding formulas in the case of parametric families of distributions have been derived, and they have played a pivotal role when establishing parametric statistical inference in the area. Non-parametric inference results have also been derived in special cases such as the tail conditional expectation, distortion risk measure, and several members of the class of weighted premiums. For weighted allocation rules, however, non-parametric inference …
We give a general result concerning the rates of convergence of penalized empirical risk minimizers (PERM) in the regression model. Then, we consider the problem of agnostic learning of the regression, and give in this context an oracle inequality and a lower bound for PERM over a finite class. These results hold for a general multivariate random design, the only assumption being the compactness of the support of its law (allowing discrete distributions for instance). Then, using these results, we construct adaptive estimators. We consider as examples adaptive estimation over anisotropic Besov…
We propose nonparametric estimators for conditional value-at-risk (CVaR) and conditional expected shortfall (CES) associated with conditional distributions of a series of returns on a financial asset. The return series and the conditioning covariates, which may include lagged returns and other exogenous variables, are assumed to be strong mixing and follow a nonparametric conditional location-scale model. First stage nonparametric estimators for location and scale are combined with a generalized Pareto approximation for distribution tails proposed by Pickands (1975) to give final estimators fo…
When simulating a complex stochastic system, the behavior of output response depends on input parameters estimated from finite real-world data, and the finiteness of data brings input uncertainty into the system. The quantification of the impact of input uncertainty on output response has been extensively studied. Most of the existing literature focuses on providing inferences on the mean response at the true but unknown input parameter, including point estimation and confidence interval construction. Risk quantification of mean response under input uncertainty often plays an important role in…
In this paper, a competing risks model is analyzed based on improved adaptive type-II progressive censored sample (IAT-II PCS). Two independent competing causes of failures are considered. It is assumed that lifetimes of the competing causes of failure follow exponential distributions with different means. Maximum likelihood estimators (MLEs) for the unknown model parameters are obtained. Using asymptotic normality property of MLE, the asymptotic confidence intervals are constructed. Existence and uniqueness properties of the MLEs are studied. Further, bootstrap confidence intervals are comput…
This paper studies $\ell_1$ regularization with high-dimensional features for support vector machines with a built-in reject option (meaning that the decision of classifying an observation can be withheld at a cost lower than that of misclassification). The procedure can be conveniently implemented as a linear program and computed using standard software. We prove that the minimizer of the penalized population risk favors sparse solutions and show that the behavior of the empirical risk minimizer mimics that of the population risk minimizer. We also introduce a notion of classification complex…
The generalization ability of minimizers of the empirical risk in the context of binary classification has been investigated under a wide variety of complexity assumptions for the collection of classifiers over which optimization is performed. In contrast, the vast majority of the works dedicated to this issue stipulate that the training dataset used to compute the empirical risk functional is composed of i.i.d. observations. Beyond the cases where training data are drawn uniformly without replacement among a large i.i.d. sample or modelled as a realization of a weakly dependent sequence of r.…
This article develops a general theory for minimum norm interpolating estimators and regularized empirical risk minimizers (RERM) in linear models in the presence of additive, potentially adversarial, errors. In particular, no conditions on the errors are imposed. A quantitative bound for the prediction error is given, relating it to the Rademacher complexity of the covariates, the norm of the minimum norm interpolator of the errors and the size of the subdifferential around the true parameter. The general theory is illustrated for Gaussian features and several norms: The $\ell_1$, $\ell_2$, g…
The cumulative incidence is the probability of failure from the cause of interest over a certain time period in the presence of other risks. A semiparametric regression model proposed by Fine and Gray (1999) has become the method of choice for formulating the effects of covariates on the cumulative incidence. Its estimation, however, requires modeling of the censoring distribution and is not statistically efficient. In this paper, we present a broad class of semiparametric transformation models which extends the Fine and Gray model, and we allow for unknown causes of failure. We derive the non…
Relative error estimation has been recently used in regression analysis. A crucial issue of the existing relative error estimation procedures is that they are sensitive to outliers. To address this issue, we employ the $γ$-likelihood function, which is constructed through $γ$-cross entropy with keeping the original statistical model in use. The estimating equation has a redescending property, a desirable property in robust statistics, for a broad class of noise distributions. To find a minimizer of the negative $γ$-likelihood function, a majorize-minimization (MM) algorithm is constructed. The…
Interval-censored competing risks data arise when each study subject may experience an event or failure from one of several causes and the failure time is not observed exactly but rather known to lie in an interval between two successive examinations. We formulate the effects of possibly time-varying covariates on the cumulative incidence or sub-distribution function (i.e., the marginal probability of failure from a particular cause) of competing risks through a broad class of semiparametric regression models that captures both proportional and non-proportional hazards structures for the sub-d…
Readily available proxies for time of disease onset such as time of the first diagnostic code can lead to substantial risk prediction error if performing analyses based on poor proxies. Due to the lack of detailed documentation and labor intensiveness of manual annotation, it is often only feasible to ascertain for a small subset the current status of the disease by a follow up time rather than the exact time. In this paper, we aim to develop risk prediction models for the onset time efficiently leveraging both a small number of labels on current status and a large number of unlabeled observat…
Nonparametric Efficient Estimation of Dynamic Treatment Regimes with Competing Risks
University of Minnesota（明尼苏达大学）；Edward Hines Jr. VA Hospital（爱德华·海恩斯少将退伍军人医院）；Augusta University（奥古斯塔大学）；Veterans Affairs Health Care System（退伍军人保健系统）；HealthPartners Institute（健康伙伴研究所）
The celebrated Bernstein von-Mises theorem ensures that credible regions from Bayesian posterior are well-calibrated when the model is correctly-specified, in the frequentist sense that their coverage probabilities tend to the nominal values as data accrue. However, this conventional Bayesian framework is known to lack robustness when the model is misspecified or only partly specified, such as in quantile regression, risk minimization based supervised/unsupervised learning and robust estimation. To overcome this difficulty, we propose a new Bayesian inferential approach that substitutes the (m…
The central problem of quantum statistics is to devise measurement schemes for the estimation of an unknown state, given an ensemble of $n$ independent identically prepared systems. For locally quadratic loss functions, the risk of standard procedures has the usual scaling of $1/n$. However, it has been noticed that for fidelity based metrics such as the Bures distance, the risk of conventional (non-adaptive) qubit tomography schemes scales as $1/\sqrt{n}$ for states close to the boundary of the Bloch sphere. Several proposed estimators appear to improve this scaling, and our goal is to analys…
Rates of convergence for empirical risk minimizers have been well studied in the literature. In this paper, we aim to provide a complementary set of results, in particular by showing that after normalization, the risk of the empirical minimizer concentrates on a single point. Such results have been established by~\cite{chatterjee2014new} for constrained estimators in the normal sequence model. We first generalize and sharpen this result to regularized least squares with convex penalties, making use of a "direct" argument based on Borell's theorem. We then study generalizations to other loss fu…
