神经网络中的廉价正交约束：正交酉群的简单参数化
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
摘要：
我们介绍了一种新的方法来执行正交和酉约束的一阶优化。这种方法是基于一个参数化源于李群理论通过指数映射。参数化将约束优化问题转化为欧氏空间上的无约束优化问题，可以使用常见的一阶优化方法。所给出的理论结果是一般的，足以涵盖特殊的正交群，酉群，并在一般情况下，任何连通紧李群。我们讨论了如何通过一个实现技巧来有效地计算这个和其他参数化，使得在神经网络中以可忽略的运行时成本使用数值上复杂的参数化。特别是，我们将我们的结果应用于具有正交递归权重的RNN，产生了一种称为expRNN的新架构。我们展示了我们的方法如何构成一种更强大的正交约束优化方法，在几个旨在测试RNN的任务中显示出更快，更准确，更稳定的收敛性。
英文摘要：
We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
