酉权的投影算法
A Projection Algorithm for the Unitary Weights
摘要：
酉神经网络是解决爆炸和消失激活/梯度问题的有前途的替代方案，而不需要显式归一化，从而降低推理速度。然而，由于其权重矩阵上的额外酉约束，它们通常需要更长的训练时间。在这里，我们展示了一种新的算法，该算法使用李代数的反向传播技术来计算来自其预训练的非酉对应物的近似酉权重。使用这些近似初始化的酉网络可以更快地达到所需的精度，从而在保持推理加速的同时减轻其训练时间损失。我们的方法将有助于酉网络的自适应，特别是对于那些预先训练的权重可以免费获得的神经架构。
英文摘要：
Unitary neural networks are promising alternatives for solving the exploding and vanishing activation/gradient problem without the need for explicit normalization that reduces the inference speed. However, they often require longer training time due to the additional unitary constraints on their weight matrices. Here we show a novel algorithm using a backpropagation technique with Lie algebra for computing approximated unitary weights from their pre-trained, non-unitary counterparts. The unitary networks initialized with these approximations can reach the desired accuracies much faster, mitigating their training time penalties while maintaining inference speedups. Our approach will be instrumental in the adaptation of unitary networks, especially for those neural architectures where pre-trained weights are freely available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
