Deep neural networks come in many sizes and architectures. The choice of architecture, in conjunction with the dataset and learning algorithm, is commonly understood to affect the learned neural representations. Yet, recent results have shown that different architectures learn representations with striking qualitative similarities. Here we derive an effective theory of representation learning under the assumption that the encoding map from input to hidden representation and the decoding map from representation to output are arbitrary smooth functions. This theory schematizes representation learning dynamics in the regime of complex, large architectures, where hidden representations are not strongly constrained by the parametrization. We show through experiments that the effective theory describes aspects of representation learning dynamics across a range of deep networks with different activation functions and architectures, and exhibits phenomena similar to the "rich" and "lazy" regime. While many network behaviors depend quantitatively on architecture, our findings point to certain behaviors that are widely conserved once models are sufficiently flexible.
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Hong Kong University of Science and Technology（香港科技大学）；Amazon.com, Inc., Bellevue, WA（亚马逊公司）
The Umwelt Representation Hypothesis: Rethinking Universality
Institute of Cognitive Science, University of Osnabrück（奥尔登堡大学认知科学研究所）；Department of Psychology and Education, Freie Universität Berlin（柏林自由大学心理学与教育系）；Bernstein Center for Computational Neuroscience, Berlin（柏林计算神经科学中心）
Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks
Center for Advanced Intelligence Project (AIP), RIKEN（RIKEN 先进智能项目中心）；NTT Network Service Systems Laboratories, NTT Corporation（NTT 网络服务系统实验室，NTT 公司）；Center for Data Science, Ehime University（爱媛大学数据科学中心）
