CWY参数化：正交矩阵和Stiefel矩阵优化的一种求解方法
CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
摘要：
我们介绍了一种高效的方法，优化正交组高度并行计算单元，如GPU或TPU。在早期的工作中，我们参数化的正交矩阵作为产品的Householder反射。然而，为了克服低并行化能力的计算Householder反射顺序，我们建议采用一种积累方案称为紧凑的WY（或CWY）变换-一个紧凑的并行化友好的矩阵表示的一系列Householder反射。我们进一步开发了一种新的截断CWY（或T-CWY）方法，用于Stiefel流形参数化，该方法具有竞争力的复杂性，并且在GPU和TPU上计算时再次产生好处。我们证明了我们的CWY和T-CWY方法导致收敛到一个稳定点的训练目标时，再加上随机梯度下降。我们将我们的方法应用于神经机器翻译和视频预测任务中的训练递归神经网络架构。
英文摘要：
We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs. As in earlier work, we parametrize an orthogonal matrix as a product of Householder reflections. However, to overcome low parallelization capabilities of computing Householder reflections sequentially, we propose employing an accumulation scheme called the compact WY (or CWY) transform -- a compact parallelization-friendly matrix representation for the series of Householder reflections. We further develop a novel Truncated CWY (or T-CWY) approach for Stiefel manifold parametrization which has a competitive complexity and, again, yields benefits when computed on GPUs and TPUs. We prove that our CWY and T-CWY methods lead to convergence to a stationary point of the training objective when coupled with stochastic gradient descent. We apply our methods to train recurrent neural network architectures in the tasks of neural machine translation and video prediction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
