Consider linear regression where the examples are generated by an unknown distribution on $R^d\times R$. Without any assumptions on the noise, the linear least squares solution for any i.i.d. sample will typically be biased w.r.t. the least squares optimum over the entire distribution. However, we show that if an i.i.d. sample of any size k is augmented by a certain small additional sample, then the solution of the combined sample becomes unbiased. We show this when the additional sample consists of d points drawn jointly according to the input distribution that is rescaled by the squared volume spanned by the points. Furthermore, we propose algorithms to sample from this volume-rescaled distribution when the data distribution is only known through an i.i.d sample.
University of Michigan（密歇根大学）；Google Inc.（谷歌公司）；Columbia University（哥伦比亚大学）
In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over $d$-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the sample is drawn i.i.d. from the input distribution, the least squares solution for that sample can be viewed as the natural estimator of the optimum. Unfortunately, this estimator almost always incurs an undesirable bias coming from the randomness of the input points, which is a significant bottleneck in model averaging. In this paper we show that it is possib…
Tsinghua University（清华大学）；Microsoft Research（微软研究院）；University of Pittsburgh（匹兹堡大学）
This paper considers doing quantile regression on censored data using neural networks (NNs). This adds to the survival analysis toolkit by allowing direct prediction of the target variable, along with a distribution-free characterisation of uncertainty, using a flexible function approximator. We begin by showing how an algorithm popular in linear models can be applied to NNs. However, the resulting procedure is inefficient, requiring sequential optimisation of an individual NN at each desired quantile. Our major contribution is a novel algorithm that simultaneously optimises a grid of quantile…
A data set sampled from a certain population is biased if the subgroups of the population are sampled at proportions that are significantly different from their underlying proportions. Training machine learning models on biased data sets requires correction techniques to compensate for the bias. We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function. Though statistically equivalent, it has been observed that resampling outperforms reweighting when combined with stochastic gradient algorith…
University of Massachusetts（马萨诸塞大学）；Mount Holyoke College（曼荷莲学院）
Field observations form the basis of many scientific studies, especially in ecological and social sciences. Despite efforts to conduct such surveys in a standardized way, observations can be prone to systematic measurement errors. The removal of systematic variability introduced by the observation process, if possible, can greatly increase the value of this data. Existing non-parametric techniques for correcting such errors assume linear additive noise models. This leads to biased estimates when applied to generalized linear models (GLM). We present an approach based on residual functions to a…
Low rank approximation and decomposition of large matrices using error correcting codes
Department of Computer Science and Engineering, University of Minnesota, Twin Cities（计算机科学与工程系，明尼苏达大学，双城分校）；Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities（电气与计算机工程系，明尼苏达大学，双城分校）
International Computer Science Institute（国际计算机科学研究所）；University of California, Berkeley（加州大学伯克利分校）；University of Bremen（不来梅大学）；DFKI GmbH（德国人工智能研究中心）
Regression evaluation has been performed for decades. Some metrics have been identified to be robust against shifting and scaling of the data but considering the different distributions of data is much more difficult to address (imbalance problem) even though it largely impacts the comparability between evaluations on different datasets. In classification, it has been stated repeatedly that performance metrics like the F-Measure and Accuracy are highly dependent on the class distribution and that comparisons between different datasets with different distributions are impossible. We show that t…
Kyoto University（京都大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；The Institute of Statistical Mathematics（统计数学研究所）；NOMURA Asset Management Co., Ltd.（野村资产管理株式会社）
For supervised classification problems, this paper considers estimating the query's label probability through local regression using observed covariates. Well-known nonparametric kernel smoother and $k$-nearest neighbor ($k$-NN) estimator, which take label average over a ball around the query, are consistent but asymptotically biased particularly for a large radius of the ball. To eradicate such bias, local polynomial regression (LPoR) and multiscale $k$-NN (MS-$k$-NN) learn the bias term by local regression around the query and extrapolate it to the query itself. However, their theoretical op…
University of California, Merced（加州大学默塞德分校）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We present a simple algorithm for identifying and correcting real-valued noisy labels from a mixture of clean and corrupted sample points using Gaussian process regression. A heteroscedastic noise model is employed, in which additive Gaussian noise terms with independent variances are associated with each and all of the observed labels. Optimizing the noise model using maximum likelihood estimation leads to the containment of the GPR model's predictive error by the posterior standard deviation in leave-one-out cross-validation. A multiplicative update scheme is proposed for solving the maximum…
Comparison of machine learning and statistical approaches for digital elevation model (DEM) correction: interim results
University of Cape Town（开普敦大学）；University of Lagos（拉各斯大学）；Newcastle University（纽卡斯尔大学）；University of Pretoria（比勒陀利亚大学）；Cape Peninsula University of Technology（开普半岛理工大学）；University of Nigeria（尼日利亚大学）；Oregon State University（俄勒冈州立大学）
本研究对比了XGBoost、LightGBM和CatBoost等梯度提升决策树与多元线性回归在提升南非开普敦30 m Copernicus和AW3D全球DEM垂直精度方面的表现。
