In this paper we propose Structuring AutoEncoders (SAE). SAEs are neural networks which learn a low dimensional representation of data which are additionally enriched with a desired structure in this low dimensional space. While traditional Autoencoders have proven to structure data naturally they fail to discover semantic structure that is hard to recognize in the raw data. The SAE solves the problem by enhancing a traditional Autoencoder using weak supervision to form a structured latent space. In the experiments we demonstrate, that the structured latent space allows for a much more efficient data representation for further tasks such as classification for sparsely labeled data, an efficient choice of data to label, and morphing between classes. To demonstrate the general applicability of our method, we show experiments on the benchmark image datasets MNIST, Fashion-MNIST, DeepFashion2 and on a dataset of 3D human shapes.
Xiaomi AI Lab（小米人工智能实验室）；Chinese University of Hong Kong（香港中文大学）
Variational Auto-Encoder (VAE) has been widely applied as a fundamental generative model in machine learning. For complex samples like imagery objects or scenes, however, VAE suffers from the dimensional dilemma between reconstruction precision that needs high-dimensional latent codes and probabilistic inference that favors a low-dimensional latent space. By virtue of high-dimensional geometry, we propose a very simple algorithm, called Spherical Auto-Encoder (SAE), completely different from existing VAEs to address the issue. SAE is in essence the vanilla autoencoder with spherical normalizat…
Mechanistic Interpretability with Sparse Autoencoder Neural Operators
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；California Institute of Technology (Caltech)（加州理工学院（Caltech））
Continuing advances in neural interfaces have enabled simultaneous monitoring of spiking activity from hundreds to thousands of neurons. To interpret these large-scale data, several methods have been proposed to infer latent dynamic structure from high-dimensional datasets. One recent line of work uses recurrent neural networks in a sequential autoencoder (SAE) framework to uncover dynamics. SAEs are an appealing option for modeling nonlinear dynamical systems, and enable a precise link between neural activity and behavior on a single-trial basis. However, the very large parameter count and co…
Can sparse autoencoders make sense of gene expression latent variable models?
Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard（埃里克和文迪斯中心，哈佛-麻省理工Broad研究所）；Department of Computer Science, University of Copenhagen（哥本哈根大学计算机科学系）
Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
School of Engineering and Applied Science, Harvard University（哈佛大学工程与应用科学学院）；CBS-NTT Program in Physics of Intelligence, Harvard University（哈佛大学人工智能物理项目）；Physics of Artificial Intelligence Group, NTT Research, Inc., Sunnyvale, CA, USA（NTT研究公司人工智能物理组）；Kempner Institute, Harvard University（哈佛大学凯普纳研究所）
Stack autoencoder (SAE), as a representative deep network, has unique and excellent performance in feature learning, and has received extensive attention from researchers. However, existing deep SAEs focus on original samples without considering the hierarchical structural information between samples. To address this limitation, this paper proposes a new SAE model-neighbouring envelope embedded stack autoencoder ensemble (NE_ESAE). Firstly, the neighbouring sample envelope learning mechanism (NSELM) is proposed for preprocessing of input of SAE. NSELM constructs sample pairs by combining neigh…
Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit
Harvard University（哈佛大学）；Kempner Institute（凯姆纳研究所）；Department of Electrical and Computer Engineering, University of Alberta（阿尔伯塔大学电气与计算机工程系）；CBS-NTT Program in Physics of Intelligence, Harvard University（哈佛大学物理智能项目）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）；Neuroscience and Mental Health Institute, University of Alberta（阿尔伯塔大学神经科学与心理健康研究所）
