用于学习深度神经网络的正交和幂等变换
Orthogonal and Idempotent Transformations for Learning Deep Neural Networks
摘要:
恒等变换在残差网络中用作跳过连接,直接连接接近输入的卷积层和接近深度神经网络输出的卷积层,改善信息流,从而简化训练。本文介绍了两种可供选择的线性变换:正交变换和幂等变换。根据正交矩阵和幂等矩阵的定义和性质,用多个正交(相同幂等)矩阵的乘积来构成线性变换,其结果等于一个正交(幂等)矩阵,从而改善了信息流,简化了训练。一个有趣的观点是,成功本质上源于前向和后向传播中的特征重用和梯度重用,用于在流动过程中保持信息,并消除梯度消失问题,因为通过跳过连接的快速方式。我们经验证明了所提出的两种转换的有效性:类似的性能在单分支网络,甚至更优越的多分支网络相比,身份转换。
英文摘要:
Identity transformations, used as skip-connections in residual networks, directly connect convolutional layers close to the input and those close to the output in deep neural networks, improving information flow and thus easing the training. In this paper, we introduce two alternative linear transforms, orthogonal transformation and idempotent transformation. According to the definition and property of orthogonal and idempotent matrices, the product of multiple orthogonal (same idempotent) matrices, used to form linear transformations, is equal to a single orthogonal (idempotent) matrix, resulting in that information flow is improved and the training is eased. One interesting point is that the success essentially stems from feature reuse and gradient reuse in forward and backward propagation for maintaining the information during flow and eliminating the gradient vanishing problem because of the express way through skip-connections. We empirically demonstrate the effectiveness of the proposed two transformations: similar performance in single-branch networks and even superior in multi-branch networks in comparison to identity transformations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。