We propose a new method for feature learning and function estimation in supervised learning via regularised empirical risk minimisation. Our approach considers functions as expectations of Sobolev functions over all possible one-dimensional projections of the data. This framework is similar to kernel ridge regression, where the kernel is $\mathbb{E}_w ( k^{(B)}(w^\top x,w^\top x^\prime))$, with $k^{(B)}(a,b) := \min(|a|, |b|)\mathds{1}_{ab>0}$ the Brownian kernel, and the distribution of the projections $w$ is learnt. This can also be viewed as an infinite-width one-hidden layer neural network, optimising the first layer's weights through gradient descent and explicitly adjusting the non-linearity and weights of the second layer. We introduce a gradient-based computational method for the estimator, called Brownian Kernel Neural Network (BKerNN), using particles to approximate the expectation, where the positive homogeneity of the Brownian kernel \red{leads to improved robustness to local minima}. Using Rademacher complexity, we show that BKerNN's expected risk converges to the minimal risk with explicit high-probability rates of $O( \min((d/n)^{1/2}, n^{-1/6}))$ (up to logarithmic factors). Numerical experiments confirm our optimisation intuitions, and BKerNN outperforms kernel ridge regression, and favourably compares to a one-hidden layer neural network with ReLU activations in various settings and real data sets.
Latent regularization for feature selection using kernel methods in tumor classification
Institut Charles Delaunay, Universite de Technologie de Troyes(特鲁瓦理工大学夏尔·德洛纳研究所);Biomedicine Research Institute of Buenos Aires - Max Planck Partner Institute(布宜诺斯艾利斯生物医学研究所-马克斯·普朗克伙伴研究所);Universidad Tecnologica Nacional Buenos Aires(布宜诺斯艾利斯国立理工大学)
Effective regions and kernels in continuous sparse regularisation, with application to sketched mixtures
Institut Camille Jordan - IUF - Centrale Lyon(卡米尔·若当研究所 - 法国高等研究科学研究院 - 里昂中央理工学院);Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668(法国国家信息与自动化研究所,法国国家科学研究中心,里昂高等师范学院,里昂第一大学,LIP实验室);INRAE, AgroParisTech, Université Paris-Saclay, MIA-Paris(法国国家农业食品与环境研究院,巴黎农业科技学校,巴黎萨克雷大学,MIA-巴黎实验室)
Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents
Center for Intelligent Decision-Making and Machine Learning, School of Management, Xi’an Jiaotong University, Xi’an, China(智能决策与机器学习中心,管理学院,西安交通大学,西安,中国);Department of Mathematics, The University of Hong Kong, Hongkong, China(数学系,香港大学,香港,中国)
Classification of geometrical objects by integrating currents and functional data analysis. An application to a 3D database of Spanish child population
Graduate University of Advanced Studies (SOKENDAI)(高等研究生院(SOKENDAI));The Institute of Statistical Mathematics(统计数学研究所);The University of Adelaide(阿德莱德大学)