arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.17732cs.LGcs.AI
收藏

最优输入增益:全面增强前馈神经网络所需的一切

Optimal Input Gain: All You Need to Supercharge a Feed-Forward Neural Network

Quantiphi Inc(Quantiphi公司);Aptiv Advance Research Center(安波福高级研究中心);Probe Technologies(探针技术公司);Amazon(亚马逊);The University of Texas at Arlington(德克萨斯大学阿灵顿分校)

Chinmay Rane, Kanishka Tyagi, Sanjeev Malalur, Yash Shinge, Michael Manry

AI总结:

提出最优输入增益(OIG)方法,通过二阶优化寻找对角自相关矩阵来改进BP和HWO等一阶两阶段训练算法,以极低计算成本显著提升前馈网络性能,使其媲美Levenberg-Marquardt方法。

摘要:

输入的线性变换会改变原本等价的前馈网络的训练性能。然而,大多数线性变换被视为独立于实际训练的预处理操作。从等价网络出发,研究表明,使用线性变换预处理输入等价于在每次训练迭代中将负梯度矩阵与自相关矩阵相乘。本文提出了一种二阶方法来寻找在给定迭代中最大化学习的自相关矩阵。当自相关矩阵为对角矩阵时,该方法优化了输入增益。这种最优输入增益(OIG)方法被用于改进两种一阶两阶段训练算法,即反向传播(BP)和隐藏权重优化(HWO),这两种算法交替更新输入权重并求解输出权重的线性方程。结果表明,所提出的OIG方法极大地增强了一阶算法的性能,通常使其能够以远少于Levenberg-Marquardt方法的计算量与之媲美。研究还表明,HWO等价于对输入应用了白化变换的BP。HWO有效地将白化变换与学习结合起来。因此,OIG改进的HWO可能成为更复杂的深度学习架构的重要构建块。

英文摘要:

Linear transformation of the inputs alters the training performance of feed-forward networks that are otherwise equivalent. However, most linear transforms are viewed as a pre-processing operation separate from the actual training. Starting from equivalent networks, it is shown that pre-processing inputs using linear transformation are equivalent to multiplying the negative gradient matrix with an autocorrelation matrix per training iteration. Second order method is proposed to find the autocorrelation matrix that maximizes learning in a given iteration. When the autocorrelation matrix is diagonal, the method optimizes input gains. This optimal input gain (OIG) approach is used to improve two first-order two-stage training algorithms, namely back-propagation (BP) and hidden weight optimization (HWO), which alternately update the input weights and solve linear equations for output weights. Results show that the proposed OIG approach greatly enhances the performance of the first-order algorithms, often allowing them to rival the popular Levenberg-Marquardt approach with far less computation. It is shown that HWO is equivalent to BP with Whitening transformation applied to the inputs. HWO effectively combines Whitening transformation with learning. Thus, OIG improved HWO could be a significant building block to more complex deep learning architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑