arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2003.02570cs.LGstat.ML
收藏

训练-再训练:将权重的位置与其值解耦

Train-by-Reconnect: Decoupling Locations of Weights from their Values

Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院)

Yushi Qiu, Reiji Suda

摘要:

是什么让未经训练的深度神经网络(DNN)与经过训练的性能不同?通过放大训练良好的DNN中的权重,我们发现正是权重的位置保存了训练编码的大部分信息。基于这一观察结果,我们假设基于随机梯度方法训练的DNN中的权重可以分为两个维度:权重的位置及其确切值。为了评估我们的假设,我们提出了一种新的方法命名为前瞻排列(LaPerm),通过重新连接的权重训练DNN。我们以经验证明了LaPerm的多功能性,同时提供了大量证据来支持我们的假设:当初始权重随机且密集时,我们的方法显示出与常规优化器相似或更好的速度和性能,例如,亚当;当初始权值是随机和稀疏的(许多零)时,我们的方法改变了神经元连接的方式,并达到了与训练有素的完全初始化的网络相当的精度;当初始权值共享单个值时,我们的方法发现权值不可知的神经网络,其精度远远优于机会。

英文摘要:

What makes untrained deep neural networks (DNNs) different from the trained performant ones? By zooming into the weights in well-trained DNNs, we found it is the location of weights that hold most of the information encoded by the training. Motivated by this observation, we hypothesize that weights in stochastic gradient-based method trained DNNs can be separated into two dimensions: the locations of weights and their exact values. To assess our hypothesis, we propose a novel method named Lookahead Permutation (LaPerm) to train DNNs by reconnecting the weights. We empirically demonstrate the versatility of LaPerm while producing extensive evidence to support our hypothesis: when the initial weights are random and dense, our method demonstrates speed and performance similar to or better than that of regular optimizers, e.g., Adam; when the initial weights are random and sparse (many zeros), our method changes the way neurons connect and reach accuracy comparable to that of a well-trained fully initialized network; when the initial weights share a single value, our method finds weight agnostic neural network with far better-than-chance accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑