Linear neural network layers that are either equivariant or invariant to permutations of their inputs form core building blocks of modern deep learning architectures. Examples include the layers of DeepSets, as well as linear layers occurring in attention blocks of transformers and some graph neural networks. The space of permutation equivariant linear layers can be identified as the invariant subspace of a certain symmetric group representation, and recent work parameterized this space by exhibiting a basis whose vectors are sums over orbits of standard basis elements with respect to the symmetric group action. A parameterization opens up the possibility of learning the weights of permutation equivariant linear layers via gradient descent. The space of permutation equivariant linear layers is a generalization of the partition algebra, an object first discovered in statistical physics with deep connections to the representation theory of the symmetric group, and the basis described above generalizes the so-called orbit basis of the partition algebra. We exhibit an alternative basis, generalizing the diagram basis of the partition algebra, with computational benefits stemming from the fact that the tensors making up the basis are low rank in the sense that they naturally factorize into Kronecker products. Just as multiplication by a rank one matrix is far less expensive than multiplication by an arbitrary matrix, multiplication with these low rank tensors is far less expensive than multiplication with elements of the orbit basis. Finally, we describe an algorithm implementing multiplication with these basis elements.
McGill University(麦吉尔大学);Mila Quebec AI Institute(米拉魁北克人工智能研究所);RWTH Aachen University(亚琛工业大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所)
Heidelberg Institute for Theoretical Studies(海德堡理论研究所);Mathematical Institute, University of Oxford(牛津大学数学研究所);City University of Hong Kong(香港城市大学);IAR, Karlsruhe Institute of Technology(卡尔斯鲁厄技术大学IAR研究所);Department of Psychiatry, Warneford Hospital(沃内福德医院精神病科)
PERM EQ x GRAPH EQ: Equivariant Neural Networks for Quantum Molecular Learning
Independent Researcher, College Park, Maryland, United States(独立研究者,马里兰州学院公园,美国);Independent AI Researcher, San Francisco, California, United States(独立AI研究者,加利福尼亚州旧金山,美国)
P-Tensors: a General Formalism for Constructing Higher Order Message Passing Networks
Department of Computer Science(计算机科学系);University of Chicago(芝加哥大学);Computational and Applied Math Department of Statistics(计算与应用数学统计系);Department of Statistics(统计系)
Weizmann Institute of Science(魏茨曼科学研究所);Department of Computer Science and Applied Mathematics(计算机科学与应用数学系)
使用对排列不变或等变的深度神经网络来学习无序集合上的函数已经变得普遍。最流行的基本模型是DeepSets [Zaheer et al. 2017]和PointNet [Qi et al. 2017]。虽然已知DeepSets和PointNet对于近似不变函数是通用的,但在近似\emph{equivariant}集合函数时并不通用。另一方面,最近的几个等变集架构已被证明是等变泛的[Sannai et al. 2019],[Keriven et al. 2019],然而这些模型要么使用不是置换等变的层(在标准意义上)和/或使用较不实用的高阶张量变量。因此,在理解流行的同变集模型与理论模型的普遍性方面存在差距。在本文中,我们通过证明:(i)PointNet不是等变通用的;(ii)添加单个线性传输层使PointNet通用。我们称这种架构PointNetST,并认为它是迄今为止已知的最简单的置换等变通用模型。另一个结果是DeepSets是通用的,PointNetSeg,一种流行的点云分割网络(例如,在[Qi et al. 2017]中使用)也是通用的。用于证明上述结果的关键理论工具是所有置换等变多项式层的显式表征。最后,我们提供了数值实验验证理论结果和比较不同的置换等变模型。
Istituto Nazionale di Ottica del Consiglio Nazionale delle Ricerche (CNR-INO)(意大利国家光学研究所(国家研究委员会光学研究所));University of Florence(佛罗伦萨大学);European Laboratory for Non-Linear Spectroscopy (LENS)(欧洲非线性光谱实验室)
On the Expressive Power of Floating-Point Transformers
Department of Artificial Intelligence, Korea University(韩国大学人工智能系);Department of Mathematics and Statistics, Sejong University(世宗大学数学与统计学系);Department of Mathematical Sciences, Gwangju Institute of Science and Technology(全州科学技术研究院数学科学系)