我们建立了一个面向连续回归问题中协变量未观测尾部的外推（或域外泛化）的统计学习理论框架。我们的策略是对具有连续标签且距离原点最远的观测值子样本执行统计回归，并特别关注其角度分量。我们方法的底层假设基于多元正则变化理论，这是极值理论的基石。我们处理了从 Vapnik-Chervonenkis class 中选择预测变量的非参数最小二乘回归这一典型问题。这项工作有助于为增强协变量重尾部性能的监督学习策略开发统计学习理论基础。该领域先前的努力完全集中在极值协变量上的二元分类。尽管连续目标设定需要不同的技术和正则性假设，我们的主要结果与早期研究的发现相呼应。我们以 excess risk 量化尾部区域的预测性能，将其呈现为具有清晰偏差-方差分解的有限样本风险界限。使用模拟和真实数据的数值实验说明了我们的理论发现。
英文摘要：
We establish a statistical learning theoretical framework aimed at extrapolation, or out-of-domain generalization, on the unobserved tails of covariates in continuous regression problems. Our strategy involves performing statistical regression on a subsample of observations with continuous labels that are the furthest away from the origin, focusing specifically on their angular components. The underlying assumptions of our approach are grounded in the theory of multivariate regular variation, a cornerstone of extreme value theory. We address the stylized problem of nonparametric least squares regression with predictors chosen from a Vapnik-Chervonenkis class.
This work contributes to a broader initiative to develop statistical learning theoretical foundations for supervised learning strategies that enhance performance on the supposedly heavy tails of covariates. Previous efforts in this area have focused exclusively on binary classification on extreme covariates. Although the continuous target setting necessitates different techniques and regularity assumptions, our main results echo findings from earlier studies. We quantify the predictive performance on tail regions in terms of excess risk, presenting it as a finite sample risk bound with a clear bias-variance decomposition. Numerical experiments with simulated and real data illustrate our theoretical findings.
Supervised learning by extreme learning machines resp. neural networks with random weights is studied under a non-stationary spatial-temporal sampling design which especially addresses settings where an autonomous object moving in a non-stationary spatial environment collects and analyzes data. The stochastic model especially allows for spatial heterogeneity and weak dependence. As efficient and computationally cheap learning methods (unconstrained) least squares, ridge regression and $\ell_s$-penalized least squares (including the LASSO) are studied. Consistency and asymptotic normality of th…
We revisit the model of heteroscedastic extremes initially introduced by Einmahl et al. (JRSSB, 2016) to describe the evolution of a non stationary sequence whose extremes evolve over time and adapt it into a general extreme quantile regression framework. We provide estimates for the extreme value index and the integrated skedasis function and prove their asymptotic normality. Our results are quite similar to those developed for heteroscedastic extremes but with a different proof approach emphasizing coupling arguments. We also propose a pointwise estimator of the skedasis function and a Weiss…
Aiming to estimate extreme precipitation forecast quantiles, we propose a nonparametric regression model that features a constant extreme value index. Using local linear quantile regression and an extrapolation technique from extreme value theory, we develop an estimator for conditional quantiles corresponding to extreme high probability levels. We establish uniform consistency and asymptotic normality of the estimators. In a simulation study, we examine the performance of our estimator on finite samples in comparison with a method assuming linear quantiles. On a precipitation data set in the …
In several different fields, there is interest in analyzing the upper or lower tail quantile of the underlying distribution rather than mean or center quantile. However, the investigation of the tail quantile is difficult because of data sparsity. In this paper, we attempt to develop nonparametric quantile regression for the extremal quantile level. In extremal quantile regression, there are two types of technical conditions of the order of convergence of the quantile level: intermediate order or extreme order. For the intermediate order quantile, the ordinary nonparametric estimator is used. …
Regression modelling of spatiotemporal extreme U.S. wildfires via partially-interpretable neural networks
School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Statistics Program, Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学计算机、电气与数学科学与工程（CEMSE）分部统计项目）
Generalized extreme value (GEV) regression is often more adapted when we investigate a relationship between a binary response variable $Y$ which represents a rare event and potentiel predictors $\mathbf{X}$. In particular, we use the quantile function of the GEV distribution as link function. Bootstrapping assigns measures of accuracy (bias, variance, confidence intervals, prediction error, test of hypothesis) to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods. Bootstrapping estimates the properties of an est…
We study tail risk dynamics in high-frequency financial markets and their connection with trading activity and market uncertainty. We introduce a dynamic extreme value regression model accommodating both stationary and local unit-root predictors to appropriately capture the time-varying behaviour of the distribution of high-frequency extreme losses. To characterize trading activity and market uncertainty, we consider several volatility and liquidity predictors, and propose a two-step adaptive $L_1$-regularized maximum likelihood estimator to select the most appropriate ones. We establish the o…
Extreme quantile regression provides estimates of conditional quantiles outside the range of the data. Classical quantile regression performs poorly in such cases since data in the tail region are too scarce. Extreme value theory is used for extrapolation beyond the range of observed values and estimation of conditional extreme quantiles. Based on the peaks-over-threshold approach, the conditional distribution above a high threshold is approximated by a generalized Pareto distribution with covariate dependent parameters. We propose a gradient boosting procedure to estimate a conditional genera…
Extreme value applications commonly employ regression techniques to capture cross-sectional heterogeneity or time-variation in the data. Estimation of the parameters of an extreme value regression model is notoriously challenging due to the small number of observations that are usually available in applications. When repeated extreme measurements are collected on the same individuals, i.e., a panel of extremes is available, pooling the observations in groups can improve the statistical inference. We study three data sets related to risk assessment in finance, climate science, and hydrology. In…
Recently a distribution free approach for testing parametric hypotheses based on unitary transformations has been suggested in \cite{Khm13, Khm16, Khm17} and further studied in \cite{Ngu17} and \cite{Rob19}. In this note we show that the transformation takes extremely simple form in distribution free testing of linear regression. Then we extend it to general parametric regression with vector-valued covariates.
Hydrological post-processing using quantile regression algorithms constitutes a prime means of estimating the uncertainty of hydrological predictions. Nonetheless, conventional large-sample theory for quantile regression does not apply sufficiently far in the tails of the probability distribution of the dependent variable. To overcome this limitation that could be crucial when the interest lies on flood events, hydrological post-processing through extremal quantile regression is introduced here for estimating the extreme quantiles of hydrological model's responses. In summary, the new hydrolog…
In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, th…
Logistic regression model is widely used in many studies to investigate the relationship between a binary response variable $Y$ and a set of potential predictors $\mathbf X$. The binary response may represent, for example, the occurrence of some outcome of interest ($Y=1$ if the outcome occurred and $Y=0$ otherwise). When the dependent variable $Y$ represents a rare event, the logistic regression model shows relevant drawbacks. In order to overcome these drawbacks we propose the Generalized Extreme Value (GEV) regression model. In particular, we suggest the quantile function of the GEV distrib…
Extrapolation of extreme covariates in generalized additive regression using extreme-value theory
INRAE（法国国家农业食品环境研究院）；University of Geneva（日内瓦大学）；Univ Brest, CNRS UMR 6205, Laboratoire de Mathématiques de Bretagne Atlantique（布雷斯特大学、法国国家科学研究中心第6205联合研究单位、布列塔尼大西洋数学实验室）
We propose a new prediction method for multivariate linear regression problems where the number of features is less than the sample size but the number of outcomes is extremely large. Many popular procedures, such as penalized regression procedures, require parameter tuning that is computationally untenable in such large-scale problems. We take a different approach, motivated by ideas from simultaneous estimation problems, that performs linear shrinkage on ordinary least squares parameter estimates. Our approach is extremely computationally efficient and tuning-free. We show that it can asympt…
We propose a vector generalized additive modeling framework for taking into account the effect of covariates on angular density functions in a multivariate extreme value context. The proposed methods are tailored for settings where the dependence between extreme values may change according to covariates. We devise a maximum penalized log-likelihood estimator, discuss details of the estimation procedure, and derive its consistency and asymptotic normality. The simulation study suggests that the proposed methods perform well in a wealth of simulation scenarios by accurately recovering the true c…
We propose a "small-uniform" statistic for the inference of the functional PCA estimator in a functional linear regression model. The literature has shown two extreme behaviors: on the one hand, the FPCA estimator does not converge in distribution in its norm topology; but on the other hand, the FPCA estimator does have a pointwise asymptotic normal distribution. Our statistic takes a middle ground between these two extremes: after a suitable rate normalization, our small-uniform statistic is constructed as the maximizer of a fractional programming problem of the FPCA estimator over a finite-d…
This paper studies optimal designs for linear regression models with correlated effects for single responses. We introduce the concept of rhombic design to reduce the computational complexity and find a semi-algebraic description for the D-optimality of a rhombic design via the Kiefer-Wolfowitz equivalence theorem. Subsequently, we show that the structure of an optimal rhombic design depends directly on the correlation structure of the random coefficients.
The presence of \b{eta}-convergence in European regions is an important issue to be analyzed. In this paper, we adopt a quantile regression approach in analyzing economic convergence. While previous work has performed quantile regression at the national level, we focus on 187 European NUTS2 regions for the period 1981-2009 and use spatial quantile regression to account for spatial dependence.
In data analysis problems where we are not able to rely on distributional assumptions, what types of inference guarantees can still be obtained? Many popular methods, such as holdout methods, cross-validation methods, and conformal prediction, are able to provide distribution-free guarantees for predictive inference, but the problem of providing inference for the underlying regression function (for example, inference on the conditional mean $\mathbb{E}[Y|X]$) is more challenging. In the setting where the features $X$ are continuously distributed, recent work has established that any confidence…
Despite the renewed interest in the Newey and Powell (1987) concept of expectiles in fields such as econometrics, risk management, and extreme value theory, expectile regression---or, more generally, M-quantile regression---unfortunately remains limited to single-output problems. To improve on this, we introduce hyperplane-valued multivariate M-quantiles that show strong advantages, for instance in terms of equivariance, over the various point-valued multivariate M-quantiles available in the literature. Like their competitors, our multivariate M-quantiles are directional in nature and provide …
Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院）
Understanding the complex structure of multivariate extremes is a major challenge in various fields from portfolio monitoring and environmental risk management to insurance. In the framework of multivariate Extreme Value Theory, a common characterization of extremes' dependence structure is the angular measure. It is a suitable measure to work in extreme regions as it provides meaningful insights concerning the subregions where extremes tend to concentrate their mass. The present paper develops a novel optimization-based approach to assess the dependence structure of extremes. This support ide…
Based on the novel concept of multivariate center-outward quantiles introduced recently in Chernozhukov et al. (2017) and Hallin et al. (2021), we are considering the problem of nonparametric multiple-output quantile regression. Our approach defines nested conditional center-outward quantile regression contours and regions with given conditional probability content irrespective of the underlying distribution; their graphs constitute nested center-outward quantile regression tubes. Empirical counterparts of these concepts are constructed, yielding interpretable empirical regions and contours wh…
Construction of valid statistical inference for estimators based on data-driven selection has received a lot of attention in the recent times. Berk et al. (2013) is possibly the first work to provide valid inference for Gaussian homoscedastic linear regression with fixed covariates under arbitrary covariate/variable selection. The setting is unrealistic and is extended by Bachoc et al. (2016) by relaxing the distributional assumptions. A major drawback of the aforementioned works is that the construction of valid confidence regions is computationally intensive. In this paper, we first prove th…
Estimating the tail index parameter is one of the primal objectives in extreme value theory. For heavy-tailed distributions the Hill estimator is the most popular way to estimate the tail index parameter. Improving the Hill estimator was aimed by recent works with different methods, for example by using bootstrap, or Kolmogorov-Smirnov metric. These methods are asymptotically consistent, but for tail index $ξ>1$ and smaller sample sizes the estimation fails to approach the theoretical value for realistic sample sizes. In this paper, we introduce a new empirical method, which can estimate high …
