Autoencoders are among the earliest introduced nonlinear models for unsupervised learning. Although they are widely adopted beyond research, it has been a longstanding open problem to understand mathematically the feature extraction mechanism that trained nonlinear autoencoders provide.
In this work, we make progress in this problem by analyzing a class of two-layer weight-tied nonlinear autoencoders in the mean field framework. Upon a suitable scaling, in the regime of a large number of neurons, the models trained with stochastic gradient descent are shown to admit a mean field limiting dynamics. This limiting description reveals an asymptotically precise picture of feature learning by these models: their training dynamics exhibit different phases that correspond to the learning of different principal subspaces of the data, with varying degrees of nonlinear shrinkage dependent on the $\ell_{2}$-regularization and stopping time. While we prove these results under an idealized assumption of (correlated) Gaussian data, experiments on real-life data demonstrate an interesting match with the theory.
The autoencoder setup of interests poses a nontrivial mathematical challenge to proving these results. In this setup, the "Lipschitz" constants of the models grow with the data dimension $d$. Consequently an adaptation of previous analyses requires a number of neurons $N$ that is at least exponential in $d$. Our main technical contribution is a new argument which proves that the required $N$ is only polynomial in $d$. We conjecture that $N\gg d$ is sufficient and that $N$ is necessarily larger than a data-dependent intrinsic dimension, a behavior that is fundamentally different from previously studied setups.
Accelerating lensed quasar discovery and modeling with physics-informed variational autoencoders
Technical University of Munich（慕尼黑工业大学）；Max-Planck-Institut für Astrophysik（马克斯·普朗克天体物理研究所）；Università degli Studi di Milano（米兰大学）；INAF – IASF Milano（意大利国家天体物理研究所米兰分部）；Academia Sinica Institute of Astronomy and Astrophysics (ASIAA)（中央研究院天文及天文物理研究所）；Aix-Marseille Université（艾克斯-马赛大学）；American Museum of Natural History（美国自然历史博物馆）；Lehman College of the CUNY（纽约城市大学莱曼学院）
本文提出物理信息变分自编码器模型 VariLens，可在单 CPU 毫秒级完成透镜分类与 SIE 参数建模，并从约8000万源中筛选出42个待光谱确认的透镜类星体候选体。
Stable and Steerable Sparse Autoencoders with Weight Regularization
Department of Biology, University of Oxford, Oxford, UK（牛津大学生物学系）；Kavli Institute for Nanoscience Discovery, University of Oxford, Oxford, UK（牛津大学纳科学发现研究所）；Department of Chemistry, University of Oxford, Oxford, UK（牛津大学化学系）
Feedback Recurrent Autoencoder for Video Compression
Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.（高通人工智能研究院，高通技术荷兰有限公司）；Qualcomm AI Research, Qualcomm Technologies, Inc.（高通人工智能研究院，高通技术公司）；University of Oxford（牛津大学）
