Nonlinear manifold learning from unorganized data points is a very challenging unsupervised learning and data visualization problem with a great variety of applications. In this paper we present a new algorithm for manifold learning and nonlinear dimension reduction. Based on a set of unorganized data points sampled with noise from the manifold, we represent the local geometry of the manifold using tangent spaces learned by fitting an affine subspace in a neighborhood of each data point. Those tangent spaces are aligned to give the internal global coordinates of the data points with respect to the underlying manifold by way of a partial eigendecomposition of the neighborhood connection matrix. We present a careful error analysis of our algorithm and show that the reconstruction errors are of second-order accuracy. We illustrate our algorithm using curves and surfaces both in
2D/3D and higher dimensional Euclidean spaces, and 64-by-64 pixel face images with various pose and lighting conditions. We also address several theoretical and algorithmic issues for further research and improvements.
Center for Brain Science, Harvard University（哈佛大学脑科学中心）；School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）
The training dynamics and generalization properties of neural networks (NN) can be precisely characterized in function space via the neural tangent kernel (NTK). Structural changes to the NTK during training reflect feature learning and underlie the superior performance of networks outside of the static kernel regime. In this work, we seek to theoretically understand kernel alignment, a prominent and ubiquitous structural change that aligns the NTK with the target function. We first study a toy model of kernel evolution in which the NTK evolves to accelerate training and show that alignment na…
Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Microsoft Research（微软研究院）；Canada CIFAR AI chair（加拿大高级研究所加拿大人工智能主席项目）
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization path…
Tangentially Aligned Integrated Gradients for User-Friendly Explanations
School of Electrical and Mechanical Engineering, The University of Adelaide（阿德莱德大学电气与机械工程学院）；School of Computer and Mathematical Sciences, The University of Adelaide（阿德莱德大学计算机与数学科学学院）；Information Sciences Division, Defence Science and Technology Group（澳大利亚国防科学与技术集团信息科学部）
Principal Curvatures Estimation with Applications to Single Cell Data
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；Yale University（耶鲁大学）；Université de Fribourg（弗里堡大学）；Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）
Neural networks in the lazy training regime converge to kernel machines. Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel? We demonstrate that this can indeed happen due to a phenomenon we term silent alignment, which requires that the tangent kernel of a network evolves in eigenstructure while small and before the loss appreciably decreases, and grows only in overall scale afterwards. We show that such an effect takes place in homogenous neural networks with small initialization and whitened data. We provide an analytical treatment of…
Manifold hypotheses are typically used for tasks such as dimensionality reduction, interpolation, or improving classification performance. In the less common problem of manifold estimation, the task is to characterize the geometric structure of the manifold in the original ambient space from a sample. We focus on the role that tangent bundle learners (TBL) can play in estimating the underlying manifold from which data is assumed to be sampled. Since the unbounded tangent spaces natively represent a poor manifold estimate, the problem reduces to one of estimating regions in the tangent space wh…
High dimensionality poses many challenges to the use of data, from visualization and interpretation, to prediction and storage for historical preservation. Techniques abound to reduce the dimensionality of fixed-length sequences, yet these methods rarely generalize to variable-length sequences. To address this gap, we extend existing methods that rely on the use of kernels to variable-length sequences via use of the Recurrent Neural Tangent Kernel (RNTK). Since a deep neural network with ReLu activation is a Max-Affine Spline Operator (MASO), we dub our approach Max-Affine Spline Kernel (MASK)…
This paper extends robust principal component analysis (RPCA) to nonlinear manifolds. Suppose that the observed data matrix is the sum of a sparse component and a component drawn from some low dimensional manifold. Is it possible to separate them by using similar ideas as RPCA? Is there any benefit in treating the manifold as a whole as opposed to treating each local region independently? We answer these two questions affirmatively by proposing and analyzing an optimization framework that separates the sparse component from the manifold under noisy data. Theoretical error bounds are provided w…
On the Wasserstein Geodesic Principal Component Analysis of probability measures
CREST-ENSAE, IP Paris（CREST-ENSAE，IP巴黎）；CNRS, IRIT, Université de Toulouse（CNRS，IRIT，图卢兹大学）；Université Paris 1 Panthéon Sorbonne（巴黎第一大学巴黎政治学院）；ENAC, IMT, Université de Toulouse（ENAC，IMT，图卢兹大学）
Robust Tangent Space Estimation via Laplacian Eigenvector Gradient Orthogonalization
Program in Applied and Computational Mathematics, Princeton University（普林斯顿大学应用与计算数学项目）；Department of Mathematics, UC San Diego（加州大学圣地亚哥分校数学系）；Halicioğlu Data Science Institute, UC San Diego（加州大学圣地亚哥分校Halicioğlu数据科学研究所）；Department of Mathematics and Halicioğlu Data Science Institute, UC San Diego（加州大学圣地亚哥分校数学系与Halicioğlu数据科学研究所）
Mila Institute（米拉研究所）；Saint Mary’s University（圣玛丽大学）；Université de Montréal（蒙特利尔大学）
We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks. We provide convergence guarantees with rates for deep linear networks for both continuous and discrete dynamics. Additionally, we study incremental learning phenomena for shallow linear networks. Interestingly, certain specific initializations imply that negligible components are learned before the principal ones, thus potentially negatively affecting the effectiveness of such a learning algorithm; a phenomenon we classify as implicit anti-regularization. We…
Halıcıoğlu Data Science Institute, UCSD（加州大学圣迭戈分校哈利乔格鲁数据科学学院）；Computer Science and Engineering Department, UCSD（加州大学圣迭戈分校计算机科学与工程系）；Electrical and Computer Engineering Department, UCSD（加州大学圣迭戈分校电气与计算机工程系）
