This article proposes a new method of truncated estimation to estimate the tail index $α$ of the extremely heavy-tailed distribution with infinite mean or variance. We not only present two truncated estimators $\hatα$ and $\hatα^{\prime}$ for estimating $α$ ($0<α\leq 1$) and $α$ ($1<α\leq 2$) respectively, but also prove their asymptotic statistical properties. The numerical simulation results comparing the six known estimators in estimating error, the Type I Error and the power of estimator show that the performance of the two new truncated estimators is quite good on the whole.
This article deals with the hypothesis test for the extremely heavy-tailed distributions with infinite mean or variance by using a truncated sample mean. We obtain three necessary and sufficient conditions under which the asymptotic distribution of the truncated test statistics converges to normal, neither normal nor stable or converges to $-\infty$ or the combination of stable distributions, respectively. The numerical simulation illustrates an application of the theoretical results above in the hypothesis testing.
The objective of this work is to propose a new algorithm to fit a sphere on a noisy 3D point cloud distributed around a complete or a truncated sphere. More precisely, we introduce a projected Robbins-Monro algorithm and its averaged version for estimating the center and the radius of the sphere. We give asymptotic results such as the almost sure convergence of these algorithms as well as the asymptotic normality of the averaged algorithm. Furthermore, some non-asymptotic results will be given, such as the rates of convergence in quadratic mean. Some numerical experiments show the efficiency o…
Recently some papers, such as Aban, Meerschaert and Panorska (2006), Nuyts (2010) and Clark (2013), have drawn attention to possible truncation in Pareto tail modelling. Sometimes natural upper bounds exist that truncate the probability tail, such as the Maximum Possible Loss in insurance treaties. At other instances ultimately at the largest data, deviations from a Pareto tail behaviour become apparent. This matter is especially important when extrapolation outside the sample is required. Given that in practice one does not always know whether the distribution is truncated or not, we consider…
A weighted Gaussian approximation to tail product-limit process for Pareto-like distributions of randomly right-truncated data is provided and a new consistent and asymptotically normal estimator of the extreme value index is derived. A simulation study is carried out to evaluate the finite sample behavior of the proposed estimator.
It was shown that when one disposes of a parametric information of the truncation distribution, the semiparametric estimator of the distribution function for truncated data (Wang, 1989) is more efficient than the nonparametric one. On the basis of this estimation method, we derive an estimator for the tail index of Pareto-type distributions that are randomly right-truncated and establish its consistency and asymptotic normality. The finite sample behavior of the proposed estimator is carried out by simulation study. We point out that, in terms of both bias and root of the mean squared error, o…
By means of a Lynden-Bell integral with deterministic threshold, Worms and Worms [A Lynden-Bell integral estimator for extremes of randomly truncated data. Statist. Probab. Lett. 2016; 109: 106-117] recently introduced an asymptotically normal estimator of the tail index for randomly right-truncated Pareto-type data. In this context, we consider the random threshold case to derive a Hill-type estimator and establish its consistency and asymptotic normality. A simulation study is carried out to evaluate the finite sample behavior of the proposed estimator.
In this paper, we focus on multivariate doubly truncated first two moments of generalized skew-elliptical (GSE) distributions and derive explicit expressions for them. It includes many useful distributions, for examples, generalized skew-normal (GSN), generalized skew-Laplace (GSLa), generalized skew-logistic (GSLo) and generalized skew student-$t$ (GSSt) distributions, all as special cases. We also give formulas of multivariate doubly truncated expectation and covariance for GSE distributions. As applications, we show the results of multivariate tail conditional expectation (MTCE) and multiva…
In this paper, we define a kernel estimator for the tail index of a Pareto-type distribution under random right-truncation and establish its asymptotic normality. A simulation study shows that, compared to the estimators recently proposed by Gardes & Stupfler (2015) and Benchaira et al. (2015), this newly introduced estimator behaves better, in terms of bias and mean squared error, for small samples.
This work deals with the estimation of the extreme value index and extreme quantiles for heavy tailed data,randomly right truncated by another heavy tailed variable. Under mild assumptions and the condition thatthe truncated variable is less heavy-tailed than the truncating variable, asymptotic normality is proved for bothestimators. The proposed estimator of the extreme value index is an adaptation of the Hill estimator, in thenatural form of a Lynden-Bell integral. Simulations illustrate the quality of the estimators under a variety ofsituations.
Recently attention has been drawn to practical problems with the use of unbounded Pareto distributions, for instance when there are natural upper bounds that truncate the probability tail. Aban, Meerschaert and Panorska (2006) derived the maximum likelihood estimator for the Pareto tail index of a truncated Pareto distribution with a right truncation point $T$. The Hill (1975) estimator is then obtained by letting $T \to \infty$. The problem of extreme value estimation under right truncation was also introduced in Nuyts (2010) who proposed a similar estimator for the tail index and considered …
University of Bristol（布里斯托大学）；Tokyo Institute of Technology（东京工业大学）；RIKEN AIP（理化学研究所人工智能研究中心）
Truncated densities are probability density functions defined on truncated domains. They share the same parametric form with their non-truncated counterparts up to a normalizing constant. Since the computation of their normalizing constants is usually infeasible, Maximum Likelihood Estimation cannot be easily applied to estimate truncated density models. Score Matching (SM) is a powerful tool for fitting parameters using only unnormalized models. However, it cannot be directly applied here as boundary conditions used to derive a tractable SM objective are not satisfied by truncated densities. …
The problem of estimating the tail index from truncated data is addressed in Chakrabarty and Samorodnitsky (2009). In that paper, a sample based (and hence random) choice of k is suggested, and it is shown that the choice leads to a consistent estimator of the inverse of the tail index. In this paper, the second order behavior of the Hill estimator with that choice of k is studied, under some additional assumptions. In the untruncated situation, it is well known that asymptotic normality of the Hill estimator follows from the assumption of second order regular variation of the underlying distr…
In this paper, we propose an estimator of the second-order parameter of randomly right-truncated Pareto-type distributions data and establish its consistency and asymptotic normality. Moreover, we derive an asymptotically unbiased estimator of the tail index and study its asymptotic behaviour. Our considerations are based on a useful Gaussian approximation of the tail product-limit process recently given by Benchaira et al. [Tail product-limit process for truncated data with application to extreme value index estimation. Extremes, 2016; 19: 219-251] and the results of Gomes et al. [Semi-parame…
The paper deals with the problem of estimating the M$^2$ (i.e. multivariate and multidimensional) spectral density function of a stationary random process or random field. We propose the $f$-truncated periodogram, i.e. a truncated periodogram where the truncation point is a suitable function $f$ of the sample size. We discuss the asymptotic consistency of the estimator and we provide three concrete problems that can be solved using the proposed approach. Simulation results show the effectiveness of the procedure.
We study random series priors for estimating a functional parameter (f\in L^2[0,1]). We show that with a series prior with random truncation, Gaussian coefficients, and inverse gamma multiplicative scaling, it is possible to achieve posterior contraction at optimal rates and adaptation to arbitrary degrees of smoothness. We present general results that can be combined with existing rate of contraction results for various nonparametric estimation problems. We give concrete examples for signal estimation in white noise and drift estimation for a one-dimensional SDE.
Louisiana State University（路易斯安那州立大学）；Southern University and A&M College（南方大学及农工学院）
In this paper, we have established a new framework of truncated inverse sampling for estimating mean values of non-negative random variables such as binomial, Poisson, hyper-geometrical, and bounded variables. We have derived explicit formulas and computational methods for designing sampling schemes to ensure prescribed levels of precision and confidence for point estimators. Moreover, we have developed interval estimation methods.
We introduce a consistent estimator of the extreme value index under random truncation based on a single sample fraction of top observations from truncated and truncation data. We establish the asymptotic normality of the proposed estimator by making use of the weighted tail-copula process framework and we check its finite sample behavior through some simulations. As an application, we provide asymptotic normality results for an estimator of the excess-of-loss reinsurance premium.
We show that the full-sample bootstrap is asymptotically valid for constructing confidence intervals for high-quantiles, tail probabilities, and other tail parameters of a univariate distribution. This resolves the doubts that have been raised about the validity of such bootstrap methods. In our extensive simulation study, the overall performance of the bootstrap method was better than that of the standard asymptotic method, indicating that the bootstrap method is at least as good, if not better than, the asymptotic method for inference. This paper also lays the foundation for developing boots…
The problem of estimating the ratio of the means of a two-component Poisson mixture model is considered, when each component is subject to zero-inflation, i.e., excess zero counts. The. resulting {\it zero-inflated Poisson mixture (ZIPM) model} can be treated as a three-component Poisson mixture model with one degenerate component. The EM algorithm is applied to obtain frequentist estimators and their standard errors, the latter determined via an explicit expression for the observed information matrix. Bayes and empirical Bayes estimators also are obtained by means of conjugate priors and thei…
Based on suitable left-truncated or censored data, two flexible classes of $M$-estimations of Weibull tail coefficient are proposed with two additional parameters bounding the impact of extreme contamination. Asymptotic normality with $\sqrt {n}$-rate of convergence is obtained. Its robustness is discussed via its asymptotic relative efficiency and influence function. It is further demonstrated by a small scale of simulations and an empirical study on CRIX.
In this paper, a novel approach to the problem of estimating the heavy-tail exponent alpha>0 of a distribution is proposed. It is based on the fact that block-maxima of size m of the independent and identically distributed data scale at a rate of m^{1/alpha}. This scaling rate can be captured well by the max-spectrum plot of the data that leads to regression based estimators. Consistency and asymptotic normality of these estimators is established under mild conditions on the behavior of the tail of the distribution. The results are obtained by establishing bounds on the rate of convergence of …
Estimating the tail index of Pareto-type distributions from geometric records
Universidad de Zaragoza（萨拉戈萨大学）；Instituto de Biocomputación y Física de Sistemas Complejos (BIFI), Universidad de Zaragoza（萨拉戈萨大学生物计算与复杂系统物理研究所 (BIFI)）；Universidad de Chile（智利大学）
We study the problem of estimating the parameters of a Gaussian distribution when samples are only shown if they fall in some (unknown) subset $S \subseteq \R^d$. This core problem in truncated statistics has long history going back to Galton, Lee, Pearson and Fisher. Recent work by Daskalakis et al. (FOCS'18), provides the first efficient algorithm that works for arbitrary sets in high dimension when the set is known, but leaves as an open problem the more challenging and relevant case of unknown truncation set. Our main result is a computationally and sample efficient algorithm for estimatin…
While the {estimation} of risk is an important question in the daily business of banking and insurance, many existing plug-in estimation procedures suffer from an unnecessary bias. This often leads to the underestimation of risk and negatively impacts backtesting results, especially in small sample cases. In this article we show that the link between estimation bias and backtesting can be traced back to the dual relationship between risk measures and the corresponding performance measures, and discuss this in reference to value-at-risk, expected shortfall and expectile value-at-risk. Motivated…
A novel statistical method is proposed and investigated for estimating a heavy tailed density under mild smoothness assumptions. Statistical analyses of heavy-tailed distributions are susceptible to the problem of sparse information in the tail of the distribution getting washed away by unrelated features of a hefty bulk. The proposed Bayesian method avoids this problem by incorporating smoothness and tail regularization through a carefully specified semiparametric prior distribution, and is able to consistently estimate both the density function and its tail index at near minimax optimal rate…
Massachusetts Institute of Technology（麻省理工学院）；Max Plank Institute for Informatics（马克斯·普朗克信息研究所）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
We provide an efficient algorithm for the classical problem, going back to Galton, Pearson, and Fisher, of estimating, with arbitrary accuracy the parameters of a multivariate normal distribution from truncated samples. Truncated samples from a $d$-variate normal ${\cal N}(\mathbfμ,\mathbfΣ)$ means a samples is only revealed if it falls in some subset $S \subseteq \mathbb{R}^d$; otherwise the samples are hidden and their count in proportion to the revealed samples is also hidden. We show that the mean $\mathbfμ$ and covariance matrix $\mathbfΣ$ can be estimated with arbitrary accuracy in polyn…
In this paper, we study the estimation of a rank-one spiked tensor in the presence of heavy tailed noise. Our results highlight some of the fundamental similarities and differences in the tradeoff between statistical and computational efficiencies under heavy tailed and Gaussian noise. In particular, we show that, for $p$ th order tensors, the tradeoff manifests in an identical fashion as the Gaussian case when the noise has finite $4(p-1)$ th moment. The difference in signal strength requirements, with or without computational constraints, for us to estimate the singular vectors at the optima…
We consider the problem of estimating the tail index $α$ of a distribution satisfying a $(α, β)$ second-order Pareto-type condition, where βis the second-order coefficient. When $β$ is available, it was previously proved that $α$ can be estimated with the oracle rate $n^{-β/(2β+1)}$. On the contrary, when $β$ is not available, estimating $α$ with the oracle rate is challenging; so additional assumptions that imply the estimability of $β$ are usually made. In this paper, we propose an adaptive estimator of $α$, and show that this estimator attains the rate $(n/\log\log n)^{-β/(2β+1)}$ without a…
We study symmetric spiked matrix models with respect to a general class of noise distributions. Given a rank-1 deformation of a random noise matrix, whose entries are independently distributed with zero mean and unit variance, the goal is to estimate the rank-1 part. For the case of Gaussian noise, the top eigenvector of the given matrix is a widely-studied estimator known to achieve optimal statistical guarantees, e.g., in the sense of the celebrated BBP phase transition. However, this estimator can fail completely for heavy-tailed noise. In this work, we exhibit an estimator that works for h…
It is well-known that random-coefficient AR(1) process can have long memory depending on the index $β$ of the tail distribution function of the random coefficient, if it is a regularly varying function at unity. We discuss estimation of $β$ from panel data comprising N random-coefficient AR(1) series, each of length T. The estimator of $β$ is constructed as a version of the tail index estimator of Goldie and Smith (1987) applied to sample lag 1 autocorrelations of individual time series. Its asymptotic normality is derived under certain conditions on N, T and some parameters of our statistical…
University of Macau（澳门大学）；Zhuhai UM Science & Technology Research Institute（珠海澳大科技研究院）；Peking University（北京大学）
There has been a surge of interest in developing robust estimators for models with heavy-tailed and bounded variance data in statistics and machine learning, while few works impose unbounded variance. This paper proposes two type of robust estimators, the ridge log-truncated M-estimator and the elastic net log-truncated M-estimator. The first estimator is applied to convex regressions such as quantile regression and generalized linear models, while the other one is applied to high dimensional non-convex learning problems such as regressions via deep neural networks. Simulations and real data a…
We propose a new approach for estimating the parameters of a probability distribution. It consists on combining two new methods of estimation. The first is based on the definition of a new distance measuring the difference between variations of two distributions on a finite number of points from their support and on using this measure for estimation purposes by the method of minimum distance. For the second method, given an empirical discrete distribution, we build up an auxiliary discrete theoretical distribution having the same support of the first and depending on the same parameters of the…
Estimating the tail index parameter is one of the primal objectives in extreme value theory. For heavy-tailed distributions the Hill estimator is the most popular way to estimate the tail index parameter. Improving the Hill estimator was aimed by recent works with different methods, for example by using bootstrap, or Kolmogorov-Smirnov metric. These methods are asymptotically consistent, but for tail index $ξ>1$ and smaller sample sizes the estimation fails to approach the theoretical value for realistic sample sizes. In this paper, we introduce a new empirical method, which can estimate high …
This paper addresses the problem of estimating, in the presence of random censoring as well as competing risks, the extreme value index of the (sub)-distribution function associated to one particular cause, in the heavy-tail case. Asymptotic normality of the proposed estimator (which has the form of an Aalen-Johansen integral, and is the first estimator proposed in this context) is established. A small simulation study exhibits its performances for finite samples. Estimation of extreme quantiles of the cumulative incidence function is also addressed.
