arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.06208cs.LGmath.COmath.RTstat.ML
收藏

用划分代数快速计算置换等变层

Fast computation of permutation equivariant layers with the partition algebra

Pacific Northwest National Laboratory(太平洋西北国家实验室);University of Washington(华盛顿大学)

Charles Godfrey, Michael G. Rawson, Davis Brown, Henry Kvinge

摘要:

线性神经网络层对其输入的排列是等变的或不变的,它们形成了现代深度学习架构的核心构建块。示例包括DeepSets的层,以及在Transformers和一些图神经网络的注意力块中出现的线性层。置换等变线性层的空间可以被确定为某个对称群表示的不变子空间,最近的工作通过展示一个基来参数化这个空间,该基的向量是关于对称群作用的标准基元素的轨道之和。参数化打开了通过梯度下降学习置换等变线性层的权重的可能性。置换等变线性层空间是划分代数的推广,划分代数是统计物理中首次发现的对象,与对称群的表示理论有着深刻的联系,上面描述的基推广了划分代数的所谓轨道基。我们展示了一个替代的基础上,概括的图的基础上的分区代数,计算的好处源于这样一个事实,即张量构成的基础是低秩的意义上,他们自然分解成克罗内克产品。正如乘一个秩为1的矩阵远比乘一个任意矩阵便宜,乘这些低秩张量远比乘轨道基的元素便宜。最后,我们描述了一个算法实现乘法与这些基本元素。

英文摘要:

Linear neural network layers that are either equivariant or invariant to permutations of their inputs form core building blocks of modern deep learning architectures. Examples include the layers of DeepSets, as well as linear layers occurring in attention blocks of transformers and some graph neural networks. The space of permutation equivariant linear layers can be identified as the invariant subspace of a certain symmetric group representation, and recent work parameterized this space by exhibiting a basis whose vectors are sums over orbits of standard basis elements with respect to the symmetric group action. A parameterization opens up the possibility of learning the weights of permutation equivariant linear layers via gradient descent. The space of permutation equivariant linear layers is a generalization of the partition algebra, an object first discovered in statistical physics with deep connections to the representation theory of the symmetric group, and the basis described above generalizes the so-called orbit basis of the partition algebra. We exhibit an alternative basis, generalizing the diagram basis of the partition algebra, with computational benefits stemming from the fact that the tensors making up the basis are low rank in the sense that they naturally factorize into Kronecker products. Just as multiplication by a rank one matrix is far less expensive than multiplication by an arbitrary matrix, multiplication with these low rank tensors is far less expensive than multiplication with elements of the orbit basis. Finally, we describe an algorithm implementing multiplication with these basis elements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑