Peter J. Bickel, Bo Li
We reveal the phenomenon that ``naive'' multivariate local polynomial regression can adapt to local smooth lower dimensional structure in the sense that it achieves the optimal convergence rate for nonparametric estimation of regression functions belonging to a Sobolev space when the predictor variables live on or close to a lower dimensional manifold.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We consider the related problems of estimating the $l_2$-norm and the squared $l_2$-norm in sparse linear regression with unknown variance, as well as the problem of testing the hypothesis that the regression parameter is null under sparse alternatives with $l_2$ separation. We establish the minimax optimal rates of estimation (respectively, testing) in these three problems.
Logistic regression with unknown sizes has many important applications in biological and medical sciences. All models about this problem in the literature are parametric ones. A semiparametric regression model is proposed. This model incorporates overdispersion due to the variation of sizes, and allows general dose-response relations. An Expectation Conditional Maximization algorithm is provided to maximize the log likelihood. The bootstrap method can be used to construct confidence intervals for regression coefficients. Simulation is performed to study the behavior of the proposed model. Two …
This paper is devoted to the estimation of the shift parameter in a semiparametric regression model when the distribution of the observation times is unknown. Hence, we propose to use a stochastic algorithm which takes into account the estimation of the distribution of the observation times. We establish the almost sure convergence of our estimator and the asymptotic normality. The main result of the paper is that, with little assumptions on the regularity of the regression function, the asymptotic variance obtained is the same as when the distribution is known. In that sense, we improve the r…
We propose an extrinsic regression framework for modeling data with manifold valued responses and Euclidean predictors. Regression with manifold responses has wide applications in shape analysis, neuroscience, medical imaging and many other areas. Our approach embeds the manifold where the responses lie onto a higher dimensional Euclidean space, obtains a local regression estimate in that space, and then projects this estimate back onto the image of the manifold. Outside the regression setting both intrinsic and extrinsic approaches have been proposed for modeling i.i.d manifold-valued data. H…
Binomial data with unknown sizes often appear in biological and medical sciences. The previous methods either use the Poisson approximation or the quasi-likelihood approach. A full likelihood approach is proposed by treating unknown sizes as latent variables. This approach simplifies analysis as maximum likelihood estimation can be applied. It also facilitates us to gain a lot more insights into efficiency loss across models and estimation precision within models. Simulation assesses the performance of the proposed model. An application to the surviving jejunal crypt data is discussed. The pro…
Under a partially linear models we study a family of robust estimates for the regression parameter and the regression function when some of the predictor variables take values on a Riemannian manifold. We obtain the consistency and the asymptotic normality of the proposed estimators. Also, we consider a robust cross validation procedure to select the smoothing parameter. Simulations and application to real data show the performance of our proposal under small samples and contamination.
Binomial data with unknown sizes often appear in biological and medical sciences and are usually overdispersed. All previous methods used parametric models and only considered overdispersion due to the variation of sizes. The proposed semiparametric model considers overdispersion due to the variation of sizes and that of probabilities. By doing this, it can include variations caused by observations, missing covariates, and random measurement errors in covariates. An Expectation Conditional Maximization algorithm is provided to stabilize the loglikelihood optimization. Selecting the number of s…
Minimax Optimal Rates for Regression on Manifolds and Distributions
针对协变量与响应均位于低维流形的分布回归问题，本文推导 minimax 下界，并提出结合对抗学习和同步最小二乘的混合估计量以获得匹配上界。
Deconvolution is a statistical inverse problem to estimate the distribution of a random variable based on its noisy observations. Despite the extensive studies on the topic, deconvolution with unknown noise distribution remains as a notoriously hard problem. We propose a matrix-based viewpoint for collective deconvolution that subsumes the setup with repeated measurements as a special case. As the main result, we describe a simple algorithm that partially utilizes matrix structure to solve deconvolution problem and provide non-asymptotic error analysis for the algorithm. We show that the propo…
Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance
本文研究了随机设计与未知误差方差下的稀疏高维线性回归，证明了后验集中速率对未知稀疏度的自适应能力，并利用 fractional posterior 和 oracle inequalities 将分析扩展至模型误设情况。
This work deals with the ill-posed inverse problem of reconstructing a function $f$ given implicitly as the solution of $g = Af$, where $A$ is a compact linear operator with unknown singular values and known eigenfunctions. We observe the function $g$ and the singular values of the operator subject to Gaussian white noise with respective noise levels $\varepsilon$ and $σ$. We develop a minimax theory in terms of both noise levels and propose an orthogonal series estimator attaining the minimax rates. This estimator requires the optimal choice of a dimension parameter depending on certain chara…
Generative model for optimal density estimation on unknown manifold
提出一种生成模型，基于几何Whitney问题解法，在未知低维流形上实现极小极大最优密度估计，并对所有γ≥1的Hölder IPMs同时最优，实验验证优于或媲美WGAN和基于分数的模型。
We study the properties of nonparametric least squares regression using deep neural networks. We derive non-asymptotic upper bounds for the prediction error of the empirical risk minimizer of feedforward deep neural regression. Our error bounds achieve minimax optimal rate and significantly improve over the existing ones in the sense that they depend polynomially on the dimension of the predictor, instead of exponentially on dimension. We show that the neural regression estimator can circumvent the curse of dimensionality under the assumption that the predictor is supported on an approximate l…
An adaptive nonparametric estimation procedure is constructed for heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (oracle inequality) is obtained
An adaptive nonparametric estimation procedure is constructed for the estimation problem of heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (an oracle inequality) is constructed.
Consider a Gaussian nonparametric regression problem having both an unknown mean function and unknown variance function. This article presents a class of difference-based kernel estimators for the variance function. Optimal convergence rates that are uniform over broad functional classes and bandwidths are fully characterized, and asymptotic normality is also established. We also show that for suitable asymptotic formulations our estimators achieve the minimax rate.
This paper concerns the estimation of the regression function at a given point in nonparametric heteroscedastic models with Gaussian noise or with noise having unknown distribution. In the two cases an asymptotically efficient kernel estimator is constructed for the minimax absolute error risk.
The multivariate linear regression model is an important tool for investigating relationships between several response variables and several predictor variables. The primary interest is in inference about the unknown regression coefficient matrix. We propose multivariate bootstrap techniques as a means for making inferences about the unknown regression coefficient matrix. These bootstrapping techniques are extensions of those developed in Freedman (1981), which are only appropriate for univariate responses. Extensions to the multivariate linear regression model are made without proof. We forma…
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper (2007) for estimation of unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk. It means that the asymptotic quadratic risk for this procedure coincides with a sharp lower bound.
We study prediction intervals based on leave-one-out residuals in a linear regression model where the number of explanatory variables can be large compared to sample size. We establish uniform asymptotic validity (conditional on the training sample) of the proposed interval under minimal assumptions on the unknown error distribution and the high dimensional design. Our intervals are generic in the sense that they are valid for a large class of linear predictors used to obtain a point forecast, such as robust M-estimators, James-Stein type estimators and penalized estimators like the LASSO. The…
Consider a noisy linear observation model with an unknown permutation, based on observing $y = Π^* A x^* + w$, where $x^* \in \mathbb{R}^d$ is an unknown vector, $Π^*$ is an unknown $n \times n$ permutation matrix, and $w \in \mathbb{R}^n$ is additive Gaussian noise. We analyze the problem of permutation recovery in a random design setting in which the entries of the matrix $A$ are drawn i.i.d. from a standard Gaussian distribution, and establish sharp conditions on the SNR, sample size $n$, and dimension $d$ under which $Π^*$ is exactly and approximately recoverable. On the computational fron…
Quantile regression under dependent censoring with unknown association
针对生存数据右删失中相依性假设不符实际的问题，本文首次结合参数copula模型与增广非对称拉普拉斯分布构建分位数回归模型，证明参数性质并通过模拟与肝移植数据验证其性能。
Sharp Information-Theoretic Thresholds for Shuffled Linear Regression
本文研究了shuffled线性回归问题，证明了在次线性区域d=o(n)下，精确恢复未知置换的信噪比阈值为n^4，几乎精确恢复的阈值为n^2。
This article considers algorithmic and statistical aspects of linear regression when the correspondence between the covariates and the responses is unknown. First, a fully polynomial-time approximation scheme is given for the natural least squares optimization problem in any constant dimension. Next, in an average-case and noise-free setting where the responses exactly correspond to a linear function of i.i.d. draws from a standard multivariate normal distribution, an efficient algorithm based on lattice basis reduction is shown to exactly recover the unknown linear function in arbitrary dimen…
Harmonic Map Regression: Rate-Optimal Nonparametric Estimation on Manifolds with Topological Recovery
本文提出调和映射回归方法，通过狄利克雷能量惩罚实现流形上非参数估计，揭示拓扑恢复现象，推导出Sobolev阶s的最小最大速率，并在多个流形上验证理论有效性。
Local Linear Regression on Manifolds and its Geometric Interpretation
针对高维数据位于未知低维非线性流形上的情况，提出直接降维至流形本征维数并在切平面估计上执行局部线性回归（LLR）的方法，同时分析其几何意义并用于流形学习。
A Unified Analysis of Multi-task Functional Linear Regression Models with Manifold Constraint and Composite Quadratic Penalty
本文研究多任务函数线性回归模型，提出具有矩阵流形约束和复合二次惩罚的双重正则化通用模型，推导出统一的收敛上界并应用于降秩和图拉普拉斯正则化等特例。
With random regressors, least squares inference is robust to correlated errors with unknown correlation structure
该研究针对随机回归元下的线性回归场景，突破传统固定回归元理论框架，通过自归一化统计量的概率分析证明t统计量渐近正态性，发现误差相关在弱信号下可提升检验势，拓展了线性回归适用范围，印证了随机化对推断鲁棒性的价值。
A general theory for Gaussian mean estimation that automatically adapts to unknown sparsity under arbitrary norms is proposed. The theory is applied to produce adaptively minimax rate-optimal estimators in high dimensional regression and matrix estimation that involve no tuning parameters.
Motivated by finance and technical applications, the objective of this paper is to consider adaptive estimation of regression and density distribution based on Fourier-Legendre expansion, and construction of confidence intervals - also adaptive. The estimators are asymptotically optimal and adaptive in the sense that they can adapt to unknown smoothness.
Quantiles and Quantile Regression on Riemannian Manifolds: a measure-transportation-based approach
本文针对无边界紧黎曼流形提出基于最优传输数据驱动排序的分位数、分布函数、秩和符号定义，建立一致收敛性与分布无关性等理论性质，并重点研究方向型或环面型多重输出的分位数回归，通过大量模拟验证其有效性。
We consider an equivariant approach imposing data-driven bounds for the variances to avoid singular and spurious solutions in maximum likelihood (ML) estimation of clusterwise linear regression models. We investigate its use in the choice of the number of components and we propose a computational shortcut, which significantly reduces the computational time needed to tune the bounds on the data. In the simulation study and the two real-data applications, we show that the proposed methods guarantee a reliable assessment of the number of components compared to standard unconstrained methods, toge…
We consider semi-supervised regression when the predictor variables are drawn from an unknown manifold. A simple two step approach to this problem is to: (i) estimate the manifold geodesic distance between any pair of points using both the labeled and unlabeled instances; and (ii) apply a k nearest neighbor regressor based on these distance estimates. We prove that given sufficiently many unlabeled points, this simple method of geodesic kNN regression achieves the optimal finite-sample minimax bound on the mean squared error, as if the manifold were known. Furthermore, we show how this approac…
In the nonparametric regression setting, we construct an estimator which is a continuous function interpolating the data points with high probability, while attaining minimax optimal rates under mean squared risk on the scale of Hölder classes adaptively to the unknown smoothness.
Linear Regression with Unknown Truncation Beyond Gaussian Features
针对未知生存集的截断线性回归问题，提出首个多项式时间算法，仅要求特征向量为次高斯分布。
Minimax Additive Regression under Unknown Dependent Designs
研究未知相依设计下加性回归的极小极大速率，提出耦合光滑类与阈值化最小二乘估计，证明未知密度可达已知密度速率或由密度正则性决定。
Non-parametric Quantile Regression and Uniform Inference with Unknown Error Distribution
本文研究了协变量存在未知分布测量误差时的条件分位数回归函数非参数估计与一致推断，提出反卷积核估计量构建一致置信带，并通过模拟与实际数据验证了方法的有效性。
Motivated by models for multiway comparison data, we consider the problem of estimating a coordinate-wise isotonic function on the domain $[0, 1]^d$ from noisy observations collected on a uniform lattice, but where the design points have been permuted along each dimension. While the univariate and bivariate versions of this problem have received significant attention, our focus is on the multivariate case $d \geq 3$. We study both the minimax risk of estimation (in empirical $L_2$ loss) and the fundamental limits of adaptation (quantified by the adaptivity index) to a family of piecewise const…
The problem of error density estimation for a functional single index model with dependent errors is studied. A Bayesian method is utilized to simultaneously estimate the bandwidths in the kernel-form error density and regression function, under an autoregressive error structure. For estimating both the regression function and error density, empirical studies show that the functional single index model gives improved estimation and prediction accuracies than any nonparametric functional regression considered. Furthermore, estimation of error density facilitates the construction of prediction i…
Conformal inference for regression on Riemannian Manifolds
本文将共形推断扩展到响应变量位于黎曼流形、协变量位于欧几里得空间的回归场景，提出分布无关的非参数预测集，并证明其经验版本几乎必然渐近收敛到总体版本，通过模拟和真实数据验证了方法有效性。
This paper studies robust regression for data on Riemannian manifolds. Geodesic regression is the generalization of linear regression to a setting with a manifold-valued dependent variable and one or more real-valued independent variables. The existing work on geodesic regression uses the sum-of-squared errors to find the solution, but as in the classical Euclidean case, the least-squares method is highly sensitive to outliers. In this paper, we use M-type estimators, including the $L_1$, Huber and Tukey biweight estimators, to perform robust geodesic regression, and describe how to calculate …
We investigate the problem of classification in the presence of unknown class-conditional label noise in which the labels observed by the learner have been corrupted with some unknown class dependent probability. In order to obtain finite sample rates, previous approaches to classification with unknown class-conditional label noise have required that the regression function is close to its extrema on sets of large measure. We shall consider this problem in the setting of non-compact metric spaces, where the regression function need not attain its extrema. In this setting we determine the minim…
Max-affine regression refers to a model where the unknown regression function is modeled as a maximum of $k$ unknown affine functions for a fixed $k \geq 1$. This generalizes linear regression and (real) phase retrieval, and is closely related to convex regression. Working within a non-asymptotic framework, we study this problem in the high-dimensional setting assuming that $k$ is a fixed constant, and focus on estimation of the unknown coefficients of the affine functions underlying the model. We analyze a natural alternating minimization (AM) algorithm for the non-convex least squares object…
Vector Quantile Regression on Manifolds
本文利用最优传输和c-凹函数，将向量分位数回归推广到流形上，定义了条件向量分位数函数，支持分位数估计、回归及置信集计算，并通过实验验证了方法的有效性。
Wasserstein F-tests for Fréchet regression on Bures-Wasserstein manifolds
本文针对Bures-Wasserstein流形上的Fréchet回归问题，提出Wasserstein F检验方法，推导了估计量收敛速率与检验统计量渐近性质，通过模拟和单细胞基因表达数据验证了方法有效性。
We review recent results for high-dimensional sparse linear regression in the practical case of unknown variance. Different sparsity settings are covered, including coordinate-sparsity, group-sparsity and variation-sparsity. The emphasis is put on non-asymptotic analyses and feasible procedures. In addition, a small numerical study compares the practical performance of three schemes for tuning the Lasso estimator and some references are collected for some more general models, including multivariate regression and nonparametric regression.
We study full Bayesian procedures for sparse linear regression when errors have a symmetric but otherwise unknown distribution. The unknown error distribution is endowed with a symmetrized Dirichlet process mixture of Gaussians. For the prior on regression coefficients, a mixture of point masses at zero and continuous distributions is considered. We study behavior of the posterior with diverging number of predictors. Conditions are provided for consistency in the mean Hellinger distance. The compatibility and restricted eigenvalue conditions yield the minimax convergence rate of the regression…
This paper studies the high-dimensional mixed linear regression (MLR) where the output variable comes from one of the two linear regression models with an unknown mixing proportion and an unknown covariance structure of the random covariates. Building upon a high-dimensional EM algorithm, we propose an iterative procedure for estimating the two regression vectors and establish their rates of convergence. Based on the iterative estimators, we further construct debiased estimators and establish their asymptotic normality. For individual coordinates, confidence intervals centered at the debiased …
We study full Bayesian procedures for high-dimensional linear regression. We adopt data-dependent empirical priors introduced in [1]. In their paper, these priors have nice posterior contraction properties and are easy to compute. Our paper extend their theoretical results to the case of unknown error variance . Under proper sparsity assumption, we achieve model selection consistency, posterior contraction rates as well as Bernstein von-Mises theorem by analyzing multivariate t-distribution.
We consider the problem of mixed linear regression (MLR), where each observed sample belongs to one of $K$ unknown linear models. In practical applications, the proportions of the $K$ components are often imbalanced. Unfortunately, most MLR methods do not perform well in such settings. Motivated by this practical challenge, in this work we propose Mix-IRLS, a novel, simple and fast algorithm for MLR with excellent performance on both balanced and imbalanced mixtures. In contrast to popular approaches that recover the $K$ models simultaneously, Mix-IRLS does it sequentially using tools from rob…
Asymptotic equivalence results for nonparametric regression experiments have always assumed that the variances of the observations are known. In practice, however the variance of each observation is generally considered to be an unknown nuisance parameter. We establish an asymptotic approximation to the nonparametric regression experiment when the value of the variance is an additional parameter to be estimated or tested. This asymptotically equivalent experiment has two components: the first contains all the information about the variance and the second has all the information about the mean.…
This paper discusses a nonparametric regression model that naturally generalizes neural network models. The model is based on a finite number of one-dimensional transformations and can be estimated with a one-dimensional rate of convergence. The model contains the generalized additive model with unknown link function as a special case. For this case, it is shown that the additive components and link function can be estimated with the optimal rate by a smoothing spline that is the solution of a penalized least squares criterion.
We consider the problem of recovering of continuous multi-dimensional functions from the noisy observations over the regular grid. Our focus is at the adaptive estimation in the case when the function can be well recovered using a linear filter, which can depend on the unknown function itself. In the companion paper "Nonparametric Denoising of Signals with Unknown Local Structure, I: Oracle Inequalities" we have shown in the case when there exists an adapted time-invariant filter, which locally recovers "well" the unknown signal, there is a numerically efficient construction of an adaptive fil…
Manifold functional multiple regression model with LRD error term
本文研究具有长程依赖误差项的流形函数多重回归问题，提出广义最小二乘估计量并分析其在指定与误设场景下的渐近性质，通过多尺度残差分析验证误差的经验分布特性。
There is increasing interest in the problem of nonparametric regression with high-dimensional predictors. When the number of predictors $D$ is large, one encounters a daunting problem in attempting to estimate a $D$-dimensional surface based on limited data. Fortunately, in many applications, the support of the data is concentrated on a $d$-dimensional subspace with $d \ll D$. Manifold learning attempts to estimate this subspace. Our focus is on developing computationally tractable and theoretically supported Bayesian nonparametric regression methods in this context. When the subspace correspo…
The single-index model is a statistical model for intrinsic regression where responses are assumed to depend on a single yet unknown linear combination of the predictors, allowing to express the regression function as $ \mathbb{E} [ Y | X ] = f ( \langle v , X \rangle ) $ for some unknown \emph{index} vector $v$ and \emph{link} function $f$. Conditional methods provide a simple and effective approach to estimate $v$ by averaging moments of $X$ conditioned on $Y$, but depend on parameters whose optimal choice is unknown and do not provide generalization bounds on $f$. In this paper we propose a…
Collinearity and near-collinearity of predictors cause difficulties when doing regression. In these cases, variable selection becomes untenable because of mathematical issues concerning the existence and numerical stability of the regression coefficients, and interpretation of the coefficients is ambiguous because gradients are not defined. Using a differential geometric interpretation, in which the regression coefficients are interpreted as estimates of the exterior derivative of a function, we develop a new method to do regression in the presence of collinearities. Our regularization scheme …
We propose a procedure to handle the problem of Gaussian regression when the variance is unknown. We mix least-squares estimators from various models according to a procedure inspired by that of Leung and Barron (2007). We show that in some cases the resulting estimator is a simple shrinkage estimator. We then apply this procedure in various statistical settings such as linear regression or adaptive estimation in Besov spaces. Our results provide non-asymptotic risk bounds for the Euclidean risk of the estimator.
The paper considers the problem of estimating the parameters in a continuous time regression model with a non-Gaussian noise of pulse type. The noise is specified by the Ornstein-Uhlenbeck process driven by the mixture of a Brownian motion and a compound Poisson process. Improved estimates for the unknown regression parameters, based on a special modification of the James-Stein procedure with smaller quadratic risk than the usual least squares estimates, are proposed. The developed estimation scheme is applied for the improved parameter estimation in the discrete time regression with the autor…
Estimation of a regression function from independent and identically distributed data is considered. The $L_2$ error with integration with respect to the distribution of the predictor variable is used as the error criterion. The rate of convergence of least squares estimates based on fully connected spaces of deep neural networks with ReLU activation function is analyzed for smooth regression functions. It is shown that in case that the distribution of the predictor variable is concentrated on a manifold, these estimates achieve a rate of convergence which depends on the dimension of the manif…
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。