We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequency signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through a Fourier lens and uncover that as the bandwidth of a coordinate-MLP is enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose a simple regularization technique that can mitigate the above problem, which can be incorporated into existing networks without any architectural modifications.
Radial Müntz-Szász Networks: Neural Architectures with Learnable Power Bases for Multidimensional Singularities
African Institute for Mathematical Sciences (AIMS), Research and Innovation Centre, Rwanda(非洲数学科学研究所(AIMS),研究与创新中心,卢旺达);Graduate School of Engineering, The University of Tokyo, Japan(东京大学工学研究科,日本);Axiom Research Group Department of Applied Mathematics and Computational Science, NM-AIST, Tanzania(Axiom研究组应用数学与计算科学系,NM-AIST,坦桑尼亚)
Chair of AI in Healthcare and Medicine(人工智能在医疗与医学中的主任);Technical University of Munich(慕尼黑技术大学);TUM University Hospital(TUM大学医院);Imperial College London(伦敦帝国学院);Munich Center for Machine Learning(慕尼黑机器学习中心);Stanford Center for Artificial Intelligence in Medicine and Imaging(斯坦福大学人工智能在医学与成像中的中心);Stanford University(斯坦福大学);Department of Radiology(放射科部门)
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China(数学科学国家重点实验室,数学与系统科学研究院,中国科学院,北京100190,中国);School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院大学数学科学学院,北京100049,中国);Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China(上海数学与交叉科学研究所(SIMIS),上海,中国);Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China(浙江省系统健康科学重点实验室,生命科学学院,杭州先进研究院,中国科学院大学,中国科学院,杭州310024,中国)