Finding overcomplete latent representations of data has applications in data analysis, signal processing, machine learning, theoretical neuroscience and many other fields. In an overcomplete representation, the number of latent features exceeds the data dimensionality, which is useful when the data is undersampled by the measurements (compressed sensing, information bottlenecks in neural systems) or composed from multiple complete sets of linear features, each spanning the data space. Independent Components Analysis (ICA) is a linear technique for learning sparse latent representations, which typically has a lower computational cost than sparse coding, its nonlinear, recurrent counterpart. While well suited for finding complete representations, we show that overcompleteness poses a challenge to existing ICA algorithms. Specifically, the coherence control in existing ICA algorithms, necessary to prevent the formation of duplicate dictionary features, is ill-suited in the overcomplete case. We show that in this case several existing ICA algorithms have undesirable global minima that maximize coherence. Further, by comparing ICA algorithms on synthetic data and natural images to the computationally more expensive sparse coding solution, we show that the coherence control biases the exploration of the data manifold, sometimes yielding suboptimal solutions. We provide a theoretical explanation of these failures and, based on the theory, propose improved overcomplete ICA algorithms. All told, this study contributes new insights into and methods for coherence control for linear ICA, some of which are applicable to many other, potentially nonlinear, unsupervised learning methods.
Analysis of the Optimization Landscapes for Overcomplete Representation Learning
New York University（纽约大学）；UC Berkeley（加州大学伯克利分校）；ByteDance Inc.（字节跳动公司）；Chinese University of Hong Kong（香港中文大学）；Rutgers University（罗格斯大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Denver（丹佛大学）
fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discovery
The Cyprus Institute（塞浦路斯研究所）；University of Athens（雅典大学）；University of Oxford（牛津大学）；Archimedes AI/Athena Research Center（Archimedes AI/ Athena 研究中心）；University of Cyprus（塞浦路斯大学）
