Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Network
Wuhan University（武汉大学）；School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）；National Center for Applied Mathematics in Hubei（湖北国家应用数学中心）；Hubei Key Laboratory of Computational Science, Wuhan University（武汉大学湖北省计算科学重点实验室）
Zhao Ding, Chenguang Duan, Yuling Jiao, Jerry Zhijian Yang
We propose SDORE, a Semi-supervised Deep Sobolev Regressor, for the nonparametric estimation of the underlying regression function and its gradient. SDORE employs deep ReQU neural networks to minimize the empirical risk with gradient norm regularization, allowing the approximation of the regularization term by unlabeled data. Our study includes a thorough analysis of the convergence rates of SDORE in $L^{2}$-norm, achieving the minimax optimality. Further, we establish a convergence rate for the associated plug-in gradient estimator, even in the presence of significant domain shift. These theoretical findings offer valuable insights for selecting regularization parameters and determining the size of the neural network, while showcasing the provable advantage of leveraging unlabeled data in semi-supervised learning. To the best of our knowledge, SDORE is the first provable neural network-based approach that simultaneously estimates the regression function and its gradient, with diverse applications such as nonparametric variable selection. The effectiveness of SDORE is validated through an extensive range of numerical simulations.
Approximation Bounds for Transformer Networks with Application to Regression
School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）；Department of Mathematics, Hong Kong University of Science and Technology（香港科技大学数学系）；Research Center for Intelligent Operations Research, The Hong Kong Polytechnic University（香港理工大学智能运筹研究中心）
In this paper, we study the statistical limits in terms of Sobolev norms of gradient descent for solving inverse problem from randomly sampled noisy observations using a general class of objective functions. Our class of objective functions includes Sobolev training for kernel regression, Deep Ritz Methods (DRM), and Physics Informed Neural Networks (PINN) for solving elliptic partial differential equations (PDEs) as special cases. We consider a potentially infinite-dimensional parameterization of our model using a suitable Reproducing Kernel Hilbert Space and a continuous parameterization of …
The Institute of Statistical Mathematics（统计数学研究所）；The University of Akron（阿克伦大学）；RIKEN AIP（理化学研究所先进智能项目中心）
Optimal transport (OT) is a popular and powerful tool for comparing probability measures. However, OT suffers a few drawbacks: (i) input measures required to have the same mass, (ii) a high computational complexity, and (iii) indefiniteness which limits its applications on kernel-dependent algorithmic approaches. To tackle issues (ii)--(iii), Le et al. (2022) recently proposed Sobolev transport for measures on a graph having the same total mass by leveraging the graph structure over supports. In this work, we consider measures that may have different total mass and are supported on a graph met…
We establish optimal convergence rates for a decomposition-based scalable approach to kernel ridge regression. The method is simple to describe: it randomly partitions a dataset of size N into m subsets of equal size, computes an independent kernel ridge regression estimator for each subset, then averages the local solutions into a global predictor. This partitioning leads to a substantial reduction in computation time versus the standard approach of performing kernel ridge regression on all N samples. Our two main theorems establish that despite the computational speed-up, statistical optimal…
University of Potsdam（波茨坦大学）；Centre de Recherche INRIA de Paris（法国国家信息与自动化研究所巴黎研究中心）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）
In this work we investigate the variation of the online kernelized ridge regression algorithm in the setting of $d-$dimensional adversarial nonparametric regression. We derive the regret upper bounds on the classes of Sobolev spaces $W_{p}^β(\mathcal{X})$, $p\geq 2, β>\frac{d}{p}$. The upper bounds are supported by the minimax regret analysis, which reveals that in the cases $β> \frac{d}{2}$ or $p=\infty$ these rates are (essentially) optimal. Finally, we compare the performance of the kernelized ridge regression forecaster to the known non-parametric forecasters in terms of the regret rates a…
Precise asymptotic analysis of Sobolev training for random feature models
Center for Computational Science and Engineering, Massachusetts Institute of Technology（麻省理工学院计算科学与工程中心）；Department of Mathematics and Statistics, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校数学与统计系）；Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）
Optimal Rates and Saturation for Noiseless Kernel Ridge Regression
Institute for Advanced Algorithmic Research（先进算法研究院）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；AI for Science Institute（AI for Science研究院）
Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；School of Mathematical Sciences（数学科学学院）；Peking University（北京大学）；Department of Computer Science（计算机科学系）；University of Warwick（沃里克大学）；Industrial Engineering & Management Sciences（工业工程与管理科学）；Northwestern University（西北大学）
In this paper we study the statistical properties of Laplacian smoothing, a graph-based approach to nonparametric regression. Under standard regularity conditions, we establish upper bounds on the error of the Laplacian smoothing estimator $\widehat{f}$, and a goodness-of-fit test also based on $\widehat{f}$. These upper bounds match the minimax optimal estimation and testing rates of convergence over the first-order Sobolev class $H^1(\mathcal{X})$, for $\mathcal{X}\subseteq \mathbb{R}^d$ and $1 \leq d < 4$; in the estimation problem, for $d = 4$, they are optimal modulo a $\log n$ factor. Ad…
In this paper we study the statistical properties of Principal Components Regression with Laplacian Eigenmaps (PCR-LE), a method for nonparametric regression based on Laplacian Eigenmaps (LE). PCR-LE works by projecting a vector of observed responses ${\bf Y} = (Y_1,\ldots,Y_n)$ onto a subspace spanned by certain eigenvectors of a neighborhood graph Laplacian. We show that PCR-LE achieves minimax rates of convergence for random design regression over Sobolev spaces. Under sufficient smoothness conditions on the design density $p$, PCR-LE achieves the optimal rates for both estimation (where th…
Learning rates for least-squares regression are typically expressed in terms of $L_2$-norms. In this paper we extend these rates to norms stronger than the $L_2$-norm without requiring the regression function to be contained in the hypothesis space. In the special case of Sobolev reproducing kernel Hilbert spaces used as hypotheses spaces, these stronger norms coincide with fractional Sobolev norms between the used Sobolev space and $L_2$. As a consequence, not only the target function but also some of its derivatives can be estimated without changing the algorithm. From a technical point of v…
University of Pennsylvania（宾夕法尼亚大学）；Stanford University（斯坦福大学）；University of Washington（华盛顿大学）
We show how to take a regression function $\hat{f}$ that is appropriately ``multicalibrated'' and efficiently post-process it into an approximately error minimizing classifier satisfying a large variety of fairness constraints. The post-processing requires no labeled data, and only a modest amount of unlabeled data and computation. The computational and sample complexity requirements of computing $\hat f$ are comparable to the requirements for solving a single fair learning task optimally, but it can in fact be used to solve many different downstream fairness-constrained learning problems effi…
Recent years have witnessed an upsurge of interest in employing flexible machine learning models for instrumental variable (IV) regression, but the development of uncertainty quantification methodology is still lacking. In this work we present a novel quasi-Bayesian procedure for IV regression, building upon the recently developed kernelized IV models and the dual/minimax formulation of IV regression. We analyze the frequentist behavior of the proposed method, by establishing minimax optimal contraction rates in $L_2$ and Sobolev norms, and discussing the frequentist validity of credible balls…
On the Rate of Convergence of Kolmogorov-Arnold Network Regression Estimators
Department of Industrial Systems Engineering and Management, National University of Singapore（新加坡国立大学工业系统工程与管理系）；Department of Civil, Environmental and Geomatic Engineering, ETH Zürich（苏黎世联邦理工学院土木、环境与地理工程系）；Internet of Things Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）物联网研究组）
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University（武汉大学）；The Chinese University of Hong Kong（香港中文大学）
We propose a penalized nonparametric approach to estimating the quantile regression process (QRP) in a nonseparable model using rectifier quadratic unit (ReQU) activated deep neural networks and introduce a novel penalty function to enforce non-crossing of quantile regression curves. We establish the non-asymptotic excess risk bounds for the estimated QRP and derive the mean integrated squared error for the estimated QRP under mild smoothness and regularity conditions. To establish these non-asymptotic risk and estimation error bounds, we also develop a new error bound for approximating $C^s$ …
Konica Minolta Laboratory Europe（柯尼卡美能达欧洲实验室）；Technical University of Munich（慕尼黑工业大学）
Variational methods for revealing visual concepts learned by convolutional neural networks have gained significant attention during the last years. Being based on noisy gradients obtained via back-propagation such methods require the application of regularization strategies. We present a mathematical framework unifying previously employed regularization methods. Within this framework, we propose a novel technique based on Sobolev gradients which can be implemented via convolutions and does not require specialized numerical treatment, such as total variation regularization. The experiments perf…
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
School of Mathematical Sciences Fudan University（复旦大学数学学院）；School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC Shanghai Jiao Tong University（上海交通大学数学学院、自然科学研究院和教育部低维量子体系科学重点实验室）；School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics Fudan University（复旦大学上海当代应用数学重点实验室数学学院）
Despite the ubiquity of the Gaussian process regression model, few theoretical results are available that account for the fact that parameters of the covariance kernel typically need to be estimated from the dataset. This article provides one of the first theoretical analyses in the context of Gaussian process regression with a noiseless dataset. Specifically, we consider the scenario where the scale parameter of a Sobolev kernel (such as a Matérn kernel) is estimated by maximum likelihood. We show that the maximum likelihood estimation of the scale parameter alone provides significant adaptat…
