Random forests are powerful non-parametric regression method but are severely limited in their usage in the presence of randomly censored observations, and naively applied can exhibit poor predictive performance due to the incurred biases. Based on a local adaptive representation of random forests, we develop its regression adjustment for randomly censored regression quantile models. Regression adjustment is based on a new estimating equation that adapts to censoring and leads to quantile score whenever the data do not exhibit censoring. The proposed procedure named {\it censored quantile regression forest}, allows us to estimate quantiles of time-to-event without any parametric modeling assumption. We establish its consistency under mild model specifications. Numerical studies showcase a clear advantage of the proposed procedure.
Tsinghua University（清华大学）；Microsoft Research（微软研究院）；University of Pittsburgh（匹兹堡大学）
This paper considers doing quantile regression on censored data using neural networks (NNs). This adds to the survival analysis toolkit by allowing direct prediction of the target variable, along with a distribution-free characterisation of uncertainty, using a flexible function approximator. We begin by showing how an algorithm popular in linear models can be applied to NNs. However, the resulting procedure is inefficient, requiring sequential optimisation of an individual NN at each desired quantile. Our major contribution is a novel algorithm that simultaneously optimises a grid of quantile…
Department of Mathematics University of California San Diego（加利福尼亚大学圣地亚哥分校数学系）
Random forests are powerful non-parametric regression method but are severely limited in their usage in the presence of randomly censored observations, and naively applied can exhibit poor predictive performance due to the incurred biases. Based on a local adaptive representation of random forests, we develop its regression adjustment for randomly censored regression quantile models. Regression adjustment is based on new estimating equations that adapt to censoring and lead to quantile score whenever the data do not exhibit censoring. The proposed procedure named censored quantile regression f…
With the availability of high dimensional genetic biomarkers, it is of interest to identify heterogeneous effects of these predictors on patients' survival, along with proper statistical inference. Censored quantile regression has emerged as a powerful tool for detecting heterogeneous effects of covariates on survival outcomes. To our knowledge, there is little work available to draw inference on the effects of high dimensional predictors for censored quantile regression. This paper proposes a novel procedure to draw inference on all predictors within the framework of global censored quantile …
In many applications, heterogeneous treatment effects on a censored response variable are of primary interest, and it is natural to evaluate the effects at different quantiles (e.g., median). The large number of potential effect modifiers, the unknown structure of the treatment effects, and the presence of right censoring pose significant challenges. In this paper, we develop a hybrid forest approach called Hybrid Censored Quantile Regression Forest (HCQRF) to assess the heterogeneous effects varying with high-dimensional variables. The hybrid estimation approach takes advantage of the random …
We consider the problem of nonparametric quantile regression for twice censored data. Two new estimates are presented, which are constructed by applying concepts of monotone rearrangements to estimates of the conditional distribution function. The proposed methods avoid the problem of crossing quantile curves. Weak uniform consistency and weak convergence is established for both estimates and their finite sample properties are investigated by means of a simulation study. As a by-product, we obtain a new result regarding the weak convergence of the Beran estimator for right censored data on the…
We consider quantile regression processes from censored data under dependent data structures and derive a uniform Bahadur representation for those processes. We also consider cases where the dimension of the parameter in the quantile regression model is large. It is demonstrated that traditional penalized estimators such as the adaptive lasso yield sub-optimal rates if the coefficients of the quantile regression cross zero. New penalization techniques are introduced which are able to deal with specific problems of censored data and yield estimates with an optimal rate. In contrast to most of t…
We propose a censored quantile regression estimator motivated by unbiased estimating equations. Under the usual conditional independence assumption of the survival time and the censoring time given the covariates, we show that the proposed estimator is consistent and asymptotically normal. We develop an efficient computational algorithm which uses existing quantile regression code. As a result, bootstrap-type inference can be efficiently implemented. We illustrate the finite-sample performance of the proposed method by simulation studies and analysis of a survival data set.
This paper, investigates the conditional quantile estimation of a scalar random response and a functional random covariate (i.e. valued in some infinite-dimensional space) whenever {\it functional stationary ergodic data with random censorship} are considered. We introduce a kernel type estimator of the conditional quantile function. We establish the strong consistency with rate of this estimator as well as the asymptotic normality which induces a confidence interval that is usable in practice since it does not depend on any unknown quantity. An application to electricity peak demand interval …
The computational prediction algorithm of neural network, or deep learning, has drawn much attention recently in statistics as well as in image recognition and natural language processing. Particularly in statistical application for censored survival data, the loss function used for optimization has been mainly based on the partial likelihood from Cox's model and its variations to utilize existing neural network library such as Keras, which was built upon the open source library of TensorFlow. This paper presents a novel application of the neural network to the quantile regression for survival…
When facing multivariate covariates, general semiparametric regression techniques come at hand to propose flexible models that are unexposed to the curse of dimensionality. In this work a semiparametric copula-based estimator for conditional quantiles is investigated for complete or right-censored data. In spirit, the methodology is extending the recent work of Noh et al. (2013) and Noh et al. (2015), as the main idea consists in appropriately defining the quantile regression in terms of a multivariate copula and marginal distributions. Prior estimation of the latter and simple plug-in lead to…
Outlying observations, which significantly deviate from other measurements, may distort the conclusions of data analysis. Therefore, identifying outliers is one of the important problems that should be solved to obtain reliable results. While there are many statistical outlier detection algorithms and software programs for uncensored data, few are available for censored data. In this article, we propose three outlier detection algorithms based on censored quantile regression, two of which are modified versions of existing algorithms for uncensored or censored data, while the third is a newly d…
Censored quantile regression (CQR) has become a valuable tool to study the heterogeneous association between a possibly censored outcome and a set of covariates, yet computation and statistical inference for CQR have remained a challenge for large-scale data with many covariates. In this paper, we focus on a smoothed martingale-based sequential estimating equations approach, to which scalable gradient-based algorithms can be applied. Theoretically, we provide a unified analysis of the smoothed sequential estimator and its penalized counterpart in increasing dimensions. When the covariate dimen…
This paper proposes a new test for the comparison of conditional quantile curves when the outcome of interest, typically a duration, is subject to right censoring. The test can be applied both in the case of two independent samples and for paired data, and can be used for the comparison of quantiles at a fixed quantile level, a finite set of levels or a range of quantile levels. The asymptotic distribution of the proposed test statistics is obtained both under the null hypothesis and under local alternatives. We describe a bootstrap procedure in order to approximate the critical values, and pr…
Thus far, likelihood-based interval estimates for quantiles have not been studied in the literature on interval censored case 2 data and partly interval censored data, and, in this context, the use of smoothing has not been considered for any type of censored data. This article constructs smoothed weighted empirical likelihood ratio confidence intervals (WELRCI) for quantiles in a unified framework for various types of censored data, including right censored data, doubly censored data, interval censored data and partly interval censored data. The fourth order expansion of the weighted empirica…
Quantile regression has been advocated in survival analysis to assess evolving covariate effects. However, challenges arise when the censoring time is not always observed and may be covariate-dependent, particularly in the presence of continuously-distributed covariates. In spite of several recent advances, existing methods either involve algorithmic complications or impose a probability grid. The former leads to difficulties in the implementation and asymptotics, whereas the latter introduces undesirable grid dependence. To resolve these issues, we develop fundamental and general quantile cal…
We develop inference procedures for longitudinal data where some of the measurements are censored by fixed constants. We consider a semi-parametric quantile regression model that makes no distributional assumptions. Our research is motivated by the lack of proper inference procedures for data from biomedical studies where measurements are censored due to a fixed quantification limit. In such studies the focus is often on testing hypotheses about treatment equality. To this end, we propose a rank score test for large sample inference on a subset of the covariates. We demonstrate the importance …
We consider a regression modeling of the quantiles of residual life, remaining lifetime at a specific time. We propose a smoothed induced version of the existing non-smooth estimating equations approaches for estimating regression parameters. The proposed estimating equations are smooth in regression parameters, so solutions can be readily obtained via standard numerical algorithms. Moreover, the smoothness in the proposed estimating equations enables one to obtain a robust sandwich-type covariance estimator of regression estimators aided by an efficient resampling method. To handle data subje…
In this paper, we develop a new censored quantile instrumental variable (CQIV) estimator and describe its properties and computation. The CQIV estimator combines Powell (1986) censored quantile regression (CQR) to deal with censoring, with a control variable approach to incorporate endogenous regressors. The CQIV estimator is obtained in two stages that are non-additive in the unobservables. The first stage estimates a non-additive model with infinite dimensional parameters for the control variable, such as a quantile or distribution regression model. The second stage estimates a non-additive …
In this paper, we study a novel approach for the estimation of quantiles when facing potential right censoring of the responses. Contrary to the existing literature on the subject, the adopted strategy of this paper is to tackle censoring at the very level of the loss function usually employed for the computation of quantiles, the so-called "check" function. For interpretation purposes, a simple comparison with the latter reveals how censoring is accounted for in the newly proposed loss function. Subsequently, when considering the inclusion of covariates for conditional quantile estimation, by…
We discuss an application of Generalized Random Forests (GRF) proposed by Athey et al.(2019) to quantile regression for time series data. We extracted the theoretical results of the GRF consistency for i.i.d. data to time series data. In particular, in the main theorem, based only on the general assumptions for time series data in Davis and Nielsen (2020), and trees in Athey et al.(2019), we show that the tsQRF (time series Quantile Regression Forests) estimator is consistent. Davis and Nielsen (2020) also discussed the estimation problem using Random Forests (RF) for time series data, but the…
Reliable Real-Time Value at Risk Estimation via Quantile Regression Forest with Conformal Calibration
Institute of Statistics and Big Data, Renmin University of China（统计与大数据研究院，中国人民大学）；Department of Decision Analytics and Operations, City University of Hong Kong（决策分析与运营部门，香港城市大学）；School of Information, Renmin University of China（信息学院，中国人民大学）
To further develop the statistical inference problem for heterogeneous treatment effects, this paper builds on Breiman's (2001) random forest tree (RFT)and Wager et al.'s (2018) causal tree to parameterize the nonparametric problem using the excellent statistical properties of classical OLS and the division of local linear intervals based on covariate quantile points, while preserving the random forest trees with the advantages of constructible confidence intervals and asymptotic normality properties [Athey and Imbens (2016),Efron (2014),Wager et al.(2014)\citep{wager2014asymptotic}], we propo…
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University（武汉大学）；The Chinese University of Hong Kong（香港中文大学）
We propose a penalized nonparametric approach to estimating the quantile regression process (QRP) in a nonseparable model using rectifier quadratic unit (ReQU) activated deep neural networks and introduce a novel penalty function to enforce non-crossing of quantile regression curves. We establish the non-asymptotic excess risk bounds for the estimated QRP and derive the mean integrated squared error for the estimated QRP under mild smoothness and regularity conditions. To establish these non-asymptotic risk and estimation error bounds, we also develop a new error bound for approximating $C^s$ …
The classic censored regression model (tobit model) has been widely used in the economic literature. This model assumes normality for the error distribution and is not recommended for cases where positive skewness is present. Moreover, in regression analysis, it is well-known that a quantile regression approach allows us to study the influences of the explanatory variables on the dependent variable considering different quantiles. Therefore, we propose in this paper a quantile tobit regression model based on quantile-based log-symmetric distributions. The proposed methodology allows us to mode…
University of Patras（帕特雷大学）；Roma Tre University（罗马第三大学）；National Technical University of Athens（雅典国立技术大学）
Machine and statistical learning algorithms can be reliably automated and applied at scale. Therefore, they can constitute a considerable asset for designing practical forecasting systems, such as those related to urban water demand. Quantile regression algorithms are statistical and machine learning algorithms that can provide probabilistic forecasts in a straightforward way, and have not been applied so far for urban water demand forecasting. In this work, we aim to fill this gap by automating and extensively comparing several quantile-regression-based practical systems for probabilistic one…
We consider a simple approach for approximating detailed information about the conditional distribution of a real-valued response variable, given values for its covariates, using only the outputs from a standard regression model. We validate this approach by assessing its performance in the context of quantile regression; when applied to the outputs of linear, gradient boosted tree ensemble and random forest models. We find that it compares favourably to the standard approach for estimating quantile regression functions, especially for commonly selected tail probabilities, and is highly compet…
Forest-based methods have recently gained in popularity for non-parametric treatment effect estimation. Building on this line of work, we introduce causal survival forests, which can be used to estimate heterogeneous treatment effects in a survival and observational setting where outcomes may be right-censored. Our approach relies on orthogonal estimating equations to robustly adjust for both censoring and selection effects under unconfoundedness. In our experiments, we find our approach to perform well relative to a number of baselines.
Population quantiles and their functions are important parameters in many applications. For example, the lower quantiles often serve as crucial quality indices for forestry products. Given several independent samples from populations satisfying the density ratio model, we investigate the properties of empirical likelihood (EL) based inferences. The induced EL quantile estimators are shown to admit a Bahadur representation that leads to asymptotically valid confidence intervals for functions of quantiles. We rigorously prove that EL quantiles based on all the samples are more efficient than emp…
Rainfall ensemble forecasts have to be skillful for both low precipitation and extreme events. We present statistical post-processing methods based on Quantile Regression Forests (QRF) and Gradient Forests (GF) with a parametric extension for heavy-tailed distributions. Our goal is to improve ensemble quality for all types of precipitation events, heavy-tailed included, subject to a good overall performance. Our hybrid proposed methods are applied to daily 51-h forecasts of 6-h accumulated precipitation from 2012 to 2015 over France using the M{é}t{é}o-France ensemble prediction system called …
Texas A&M University（德克萨斯农工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；Heart, Artery, and Vein Center of Fresno（弗雷斯诺心脏、动脉和静脉中心）
Accuracies of survival models for life expectancy prediction as well as critical-care applications are significantly compromised due to the sparsity of samples and extreme imbalance between the survival (usually, the majority) and mortality class sizes. While a recent random survival forest (RSF) model overcomes the limitations of the proportional hazard assumption, an imbalance in the data results in an underestimation (overestimation) of the hazard of the mortality (survival) classes. A balanced random survival forests (BRSF) model, based on training the RSF model with data generated from a …
We propose generalized random forests, a method for non-parametric statistical estimation based on random forests (Breiman, 2001) that can be used to fit any quantity of interest identified as the solution to a set of local moment equations. Following the literature on local maximum likelihood estimation, our method considers a weighted set of nearby training examples; however, instead of using classical kernel weighting functions that are prone to a strong curse of dimensionality, we use an adaptive weighting function derived from a forest designed to express heterogeneity in the specified qu…
In this paper, nonparametric estimation of the conditional Weibull-tail coefficient when the variable of interest is right random censored is addressed. A Weissman-type estimator of conditional extreme quantile is also proposed. In addition, a simulation study is conducted to assess the finite-sample behavior of the proposed estimators and a comparison with alternative strategies is provided. Finally, the practical applicability of the methodology is presented using a real datasets of men suffering from a larynx cancer.
Estimating Distributions with Failure Rate Properties from Noisy Quantile Data
Department of Mechanical and Industrial Engineering, University of Toronto（多伦多大学机械与工业工程系）；Rotman School of Management, University of Toronto（多伦多大学罗特曼管理学院）；The Wharton School, University of Pennsylvania（宾夕法尼亚大学沃顿商学院）
College of Engineering, Mathematics and Physical Sciences, University of Exeter（埃克塞特大学工程、数学与物理科学学院）；Met Office（英国气象局）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）
Forecasting the weather is an increasingly data intensive exercise. Numerical Weather Prediction (NWP) models are becoming more complex, with higher resolutions, and there are increasing numbers of different models in operation. While the forecasting skill of NWP models continues to improve, the number and complexity of these models poses a new challenge for the operational meteorologist: how should the information from all available models, each with their own unique biases and limitations, be combined in order to provide stakeholders with well-calibrated probabilistic forecasts to use in dec…
National Research University Higher School of Economics（国立研究型大学高等经济学院）
Many economic applications including optimal pricing and inventory management requires prediction of demand based on sales data and estimation of sales reaction to a price change. There is a wide range of econometric approaches which are used to correct a bias in estimates of demand parameters on censored sales data. These approaches can also be applied to various classes of machine learning models to reduce the prediction error of sales volume. In this study we construct two ensemble models for demand prediction with and without accounting for demand censorship. Accounting for sales censorshi…
Pan African University（泛非大学）；University of Nairobi（内罗毕大学）；Michigan State University（密歇根州立大学）
A reliable and accurate forecasting model for crop yields is of crucial importance for efficient decision-making process in the agricultural sector. However, due to weather extremes and uncertainties, most forecasting models for crop yield are not reliable and accurate. For measuring the uncertainty and obtaining further information of future crop yields, a probability density forecasting model based on quantile random forest and Epanechnikov kernel function (QRF-SJ) is proposed. The nonlinear structure of random forest is applied to change the quantile regression model for building the probab…
In this paper we survey and unify a large class or $L$-functionals of the conditional distribution of the response variable in regression models. This includes robust measures of location, scale, skewness, and heavytailedness of the response, conditionally on covariates. We generalize the concepts of $L$-moments (Sittinen, 1969), $L$-skewness, and $L$-kurtosis (Hosking, 1990) and introduce order numbers for a large class of $L$-functionals through orthogonal series expansions of quantile functions. In particular, we motivate why location, scale, skewness, and heavytailedness have order numbers…
This paper addresses the problem of estimating, in the presence of random censoring as well as competing risks, the extreme value index of the (sub)-distribution function associated to one particular cause, in the heavy-tail case. Asymptotic normality of the proposed estimator (which has the form of an Aalen-Johansen integral, and is the first estimator proposed in this context) is established. A small simulation study exhibits its performances for finite samples. Estimation of extreme quantiles of the cumulative incidence function is also addressed.
