神经网络训练中的近似正交归一化
Approximated Orthonormal Normalisation in Training Neural Networks
摘要:
当采用深度学习方法来解决实际问题时,深度神经网络(DNN)的泛化是一个主要问题。在本文中,我们提出了一种新的技术,称为近似正交归一化(AON),以提高DNN模型的泛化能力。考虑来自模型中特定神经层的权重矩阵W,我们的目标是设计函数h(W),使得其行向量彼此近似正交,同时允许DNN模型足够准确地拟合训练数据。通过这样做,它将避免同一层的神经元之间的共同适应,从而能够提高网络泛化能力。具体来说,在每次迭代中,我们首先使用其泰勒展开近似(WW^T)^(-1/2),然后乘以矩阵W。之后,通过应用频谱归一化(SN)技术对矩阵乘积进行归一化以获得h(W)。从概念上讲,AON旨在将标准正交正则化转换为标准正交归一化,以避免手动平衡原始函数和惩罚函数。实验结果表明,AON产生有前途的验证性能相比,正交正规化。
英文摘要:
Generalisation of a deep neural network (DNN) is one major concern when employing the deep learning approach for solving practical problems. In this paper we propose a new technique, named approximated orthonormal normalisation (AON), to improve the generalisation capacity of a DNN model. Considering a weight matrix W from a particular neural layer in the model, our objective is to design a function h(W) such that its row vectors are approximately orthogonal to each other while allowing the DNN model to fit the training data sufficiently accurate. By doing so, it would avoid co-adaptation among neurons of the same layer to be able to improve network-generalisation capacity. Specifically, at each iteration, we first approximate (WW^T)^(-1/2) using its Taylor expansion before multiplying the matrix W. After that, the matrix product is then normalised by applying the spectral normalisation (SN) technique to obtain h(W). Conceptually speaking, AON is designed to turn orthonormal regularisation into orthonormal normalisation to avoid manual balancing the original and penalty functions. Experimental results show that AON yields promising validation performance compared to orthonormal regularisation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。