Recently, several theories including the replica method made predictions for the generalization error of Kernel Ridge Regression. In some regimes, they predict that the method has a `spectral bias': decomposing the true function $f^*$ on the eigenbasis of the kernel, it fits well the coefficients associated with the O(P) largest eigenvalues, where $P$ is the size of the training set. This prediction works very well on benchmark data sets such as images, yet the assumptions these approaches make on the data are never satisfied in practice. To clarify when the spectral bias prediction holds, we first focus on a one-dimensional model where rigorous results are obtained and then use scaling arguments to generalize and test our findings in higher dimensions. Our predictions include the classification case $f(x)=$sign$(x_1)$ with a data distribution that vanishes at the decision boundary $p(x)\sim x_1^χ$. For $χ>0$ and a Laplace kernel, we find that (i) there exists a cross-over ridge $λ^*_{d,χ}(P)\sim P^{-\frac{1}{d+χ}}$ such that for $λ\gg λ^*_{d,χ}(P)$, the replica method applies, but not for $λ\llλ^*_{d,χ}(P)$, (ii) in the ridge-less case, spectral bias predicts the correct training curve exponent only in the limit $d\rightarrow\infty$.
University of Oxford（牛津大学）；Google Research, Brain Team（谷歌研究院，大脑团队）；NYU（纽约大学）；FAIR（Facebook人工智能研究院）；BAIR（伯克利人工智能研究院）
Over the last decade, a single algorithm has changed many facets of our lives - Stochastic Gradient Descent (SGD). In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs). While SGD is guaranteed to converge to a local optimum (under loose assumptions), in some cases it may matter which local optimum is found, and this is often context-dependent. Examples frequently arise in machine learning, from shape-versus-texture-features to ensemble methods…
Tsinghua University（清华大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Many tasks in modern machine learning can be formulated as finding equilibria in \emph{sequential} games. In particular, two-player zero-sum sequential games, also known as minimax optimization, have received growing interest. It is tempting to apply gradient descent to solve minimax optimization given its popularity and success in supervised learning. However, it has been noted that naive application of gradient descent fails to find some local minimax and can converge to non-local-minimax points. In this paper, we propose \emph{Follow-the-Ridge} (FR), a novel algorithm that provably converge…
To Grok Grokking: Provable Grokking in Ridge Regression
Department of Computer Science, Purdue University, West Lafayette, IN, USA（普渡大学计算机科学系）；Department of Computer Science（计算机科学系）；Applied Mathematics, Weizmann Institute of Science, Israel（科学应用数学系，魏茨曼研究院）；Stein Faculty of Computer（斯坦因计算机科学学院）；Information Science, Ben-Gurion University of the Negev, Israel（信息科学系，本· Gurion 军事大学）
Kernel methods have had great success in Statistics and Machine Learning. Despite their growing popularity, however, less effort has been drawn towards developing kernel based classification methods on Riemannian manifolds due to difficulty in dealing with non-Euclidean geometry. In this paper, motivated by the extrinsic framework of manifold-valued data analysis, we propose a new positive definite kernel on planar Kendall shape space $Σ_2^k$, called extrinsic Veronese Whitney Gaussian kernel. We show that our approach can be extended to develop Gaussian kernels on any embedded manifold. Furth…
The large amount of online data and vast array of computing resources enable current researchers in both industry and academia to employ the power of deep learning with neural networks. While deep models trained with massive amounts of data demonstrate promising generalisation ability on unseen data from relevant domains, the computational cost of finetuning gradually becomes a bottleneck in transfering the learning to new domains. We address this issue by leveraging the low-rank property of learnt feature vectors produced from deep neural networks (DNNs) with the closed-form solution provided…
Learning dynamical systems from noisy data with Weak-form Kernel Ridge Regression
The Pennsylvania State University（宾夕法尼亚州立大学）；Department of Mathematics（数学系）；Institute for Computational and Data Sciences（计算与数据科学研究所）；Department of Aerospace Engineering（航空航天工程系）
Institute of Physics（物理研究所）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a challenge. We study this problem within a teacher-student framework for kernel regression, using `convolutional' kernels inspired by the neural tangent kernel of simple convolutional architectures of given filter size. Using heuristic methods from physics, we find in the ridgeless case that locality is key in determining the learning curve exponent $β$ (that relates the test error $ε_t\sim P^{-β}$ to the size of the trai…
LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations
Oxford Internet Institute, University of Oxford（牛津大学牛津互联网研究所）；FLAIR, University of Oxford（牛津大学FLAIR）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Cyber Security Strategic Technology Centre, ST Engineering（ST工程公司网络安全战略技术中心）
The explanation to an AI model's prediction used to support decision making in cyber security, is of critical importance. It is especially so when the model's incorrect prediction can lead to severe damages or even losses to lives and critical assets. However, most existing AI models lack the ability to provide explanations on their prediction results, despite their strong performance in most scenarios. In this work, we propose a novel explainable AI method, called PhilaeX, that provides the heuristic means to identify the optimized subset of features to form the complete explanations of AI mo…
Applied AI Lab (A2I)（应用人工智能实验室（A2I））；Dynamic Robot Systems (DRS)（动态机器人系统实验室（DRS））；Oxford Robotics Institute (ORI), University of Oxford（牛津大学牛津机器人研究所（ORI））
End-to-end visuomotor control is emerging as a compelling solution for robot manipulation tasks. However, imitation learning-based visuomotor control approaches tend to suffer from a common limitation, lacking the ability to recover from an out-of-distribution state caused by compounding errors. In this paper, instead of using tactile feedback or explicitly detecting the failure through vision, we investigate using the uncertainty of a policy neural network. We propose a novel uncertainty-based approach to detect and recover from failure cases. Our hypothesis is that policy uncertainties can i…
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
Baidu（百度）；University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；Johns Hopkins University（约翰霍普金斯大学）；Dalian University of Technology（大连理工大学）；Rutgers University（罗格斯大学）
