Optimal Input Gain: All You Need to Supercharge a Feed-Forward Neural Network
Quantiphi Inc(Quantiphi公司);Aptiv Advance Research Center(安波福高级研究中心);Probe Technologies(探针技术公司);Amazon(亚马逊);The University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Chinmay Rane, Kanishka Tyagi, Sanjeev Malalur, Yash Shinge, Michael Manry
Linear transformation of the inputs alters the training performance of feed-forward networks that are otherwise equivalent. However, most linear transforms are viewed as a pre-processing operation separate from the actual training. Starting from equivalent networks, it is shown that pre-processing inputs using linear transformation are equivalent to multiplying the negative gradient matrix with an autocorrelation matrix per training iteration. Second order method is proposed to find the autocorrelation matrix that maximizes learning in a given iteration. When the autocorrelation matrix is diagonal, the method optimizes input gains. This optimal input gain (OIG) approach is used to improve two first-order two-stage training algorithms, namely back-propagation (BP) and hidden weight optimization (HWO), which alternately update the input weights and solve linear equations for output weights. Results show that the proposed OIG approach greatly enhances the performance of the first-order algorithms, often allowing them to rival the popular Levenberg-Marquardt approach with far less computation. It is shown that HWO is equivalent to BP with Whitening transformation applied to the inputs. HWO effectively combines Whitening transformation with learning. Thus, OIG improved HWO could be a significant building block to more complex deep learning architectures.
Feed-Forward Optimization With Delayed Feedback for Neural Network Training
Scientific Computing Center (SCC), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany(科学计算中心(SCC)、卡尔斯鲁厄理工学院(KIT)、德国卡尔斯鲁厄);Helmholtz AI, Germany(亥姆霍兹人工智能、德国)
Information Theoretic Lower Bounds for Feed-Forward Fully-Connected Deep Networks
Purdue University(普渡大学);Department of Statistics(统计系);Department of Computer Science(计算机科学系);West Lafayette, IN 47906(西拉斐特,印第安纳州 47906)
在本文中,我们研究了样本复杂性下界的参数的准确恢复和一个积极的超额风险的前馈,全连接神经网络的二进制分类,使用信息理论的工具。我们证明了这些下界的生成网络的存在,其特征在于一个向后的数据生成过程中,输入生成的基础上的二进制输出,和网络参数化的隐藏层的权重参数。参数精确恢复的样本复杂度下限为$Ω(d r \log(r) + p )$,正超额风险的样本复杂度下限为$Ω(r \log(r) + p )$,其中$p$是输入的维数,$r$反映权重矩阵的秩,$d$是隐藏层的数量。据我们所知,我们的结果是第一个信息论下界。
SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials
KTH Royal Institute of Technology(瑞典皇家理工学院);INRIA Grenoble Rhone-Alpes(法国国家信息与自动化研究所格勒诺布尔罗纳-阿尔卑斯分部);University of Tokyo(东京大学);Cold Spring Harbor Laboratory(冷泉港实验室)
FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer
East China University of Science and Technology(华东理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Tencent AI Lab(腾讯人工智能实验室)