What makes untrained deep neural networks (DNNs) different from the trained performant ones? By zooming into the weights in well-trained DNNs, we found it is the location of weights that hold most of the information encoded by the training. Motivated by this observation, we hypothesize that weights in stochastic gradient-based method trained DNNs can be separated into two dimensions: the locations of weights and their exact values. To assess our hypothesis, we propose a novel method named Lookahead Permutation (LaPerm) to train DNNs by reconnecting the weights. We empirically demonstrate the versatility of LaPerm while producing extensive evidence to support our hypothesis: when the initial weights are random and dense, our method demonstrates speed and performance similar to or better than that of regular optimizers, e.g., Adam; when the initial weights are random and sparse (many zeros), our method changes the way neurons connect and reach accuracy comparable to that of a well-trained fully initialized network; when the initial weights share a single value, our method finds weight agnostic neural network with far better-than-chance accuracy.
Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks
Department of Statistics and Department of Mathematics, Stanford University(统计系和数学系,斯坦福大学);Université Paris-Saclay, CNRS, CEA, Institut de Physique Théorique, Gif-Sur-Yvette, France(巴黎-萨克雷大学,法国国家科学研究中心,法国原子能机构,理论物理研究所,法国吉夫-索-耶vette)
Mitigating Backdoors via Decoy Shortcuts and Knowledge Decoupling
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Decoupling Inherent Risk and Early Cancer Signs in Image-based Breast Cancer Risk Models
KTH Royal Institute of Technology(瑞典皇家理工学院);Science for Life Laboratory(生命科学实验室);Karolinska Institutet(卡罗林斯卡学院);Karolinska University Hospital(卡罗林斯卡大学医院)
Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networks
Osaka University(大阪大学);Curtin University(科廷大学);National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学);CyberAgent, Inc.(赛博代理公司)
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
School of Advanced Interdisciplinary Sciences, UCAS(UCAS先进交叉科学学院);MAIS, CASIA(CASIA人工智能研究所);School of Artificial Intelligence, UCAS(UCAS人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(HKISI-CAS人工智能与机器人中心);FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology(厦门理工学院计算机与信息工程学院)
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
University of Science and Technology of China(中国科学技术大学);Deep Space Exploration Laboratory(深空探测实验室);Institute for Advanced Algorithms Research(先进算法研究院);Sangfor Technologies Inc.(深信服科技股份有限公司);Research Institute of China Telecom(中国电信研究院)
Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection
School of Computing and Artificial Intelligence, Southwest Jiaotong University(西南交通大学计算机与人工智能学院);Department of Computer and Information Technology, Purdue University(普渡大学计算机与信息技术系);School of Computer Science, Chengdu University of Information Technology(成都信息工程大学计算机学院);University at Albany, State University of New York(纽约州立大学奥尔巴尼分校)