arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.19216cs.LGcond-mat.mtrl-sci
收藏

E2Former:一种具有线性缩放张量积的高效等变Transformer

E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products

HUST(华中科技大学);Microsoft Research(微软研究院)

Yunyang Li, Lin Huang, Zhihao Ding, Chu Wang, Xinran Wei, Han Yang, Zun Wang, Chang Liu, Yu Shi, Peiran Jin, Tao Qin, Mark Gerstein, Jia Zhang

AI总结:

本文提出E2Former,一种融合Wigner $6j$卷积的高效等变Transformer架构,通过将计算负担从边转移到节点,在保持模型表达能力和旋转等变性的同时大幅降低复杂度,实现了7-30倍加速,为可扩展分子建模提供了新方向。

摘要:

等变图神经网络(EGNNs)在建模化学、生物学和材料科学等微观系统方面取得了显著成功。然而,由于通过球面张量积构建边特征的高昂计算成本,EGNNs面临着重大的计算挑战,使其无法应用于大规模系统。为了解决这一局限性,我们引入了E2Former,这是一种等变且高效的transformer架构,它融合了Wigner $6j$卷积(Wigner $6j$ Conv)。通过将计算负担从边转移到节点,Wigner $6j$ Conv将复杂度从$O(|\mathcal{E}|)$降低到$O(|\mathcal{V}|)$,同时保持了模型的表达能力和旋转等变性。我们证明,与传统的$\mathrm{SO}(3)$卷积相比,该方法实现了7倍至30倍的加速。此外,我们的实验结果表明,所提出的E2Former缓解了现有方法的计算挑战,且未损害其捕获详细几何信息的能力。这一进展可能为可扩展且高效的分子建模指明一个有前景的方向。

英文摘要:

Equivariant Graph Neural Networks (EGNNs) have demonstrated significant success in modeling microscale systems, including those in chemistry, biology and materials science. However, EGNNs face substantial computational challenges due to the high cost of constructing edge features via spherical tensor products, making them impractical for large-scale systems. To address this limitation, we introduce E2Former, an equivariant and efficient transformer architecture that incorporates the Wigner $6j$ convolution (Wigner $6j$ Conv). By shifting the computational burden from edges to nodes, the Wigner $6j$ Conv reduces the complexity from $O(|\mathcal{E}|)$ to $ O(| \mathcal{V}|)$ while preserving both the model's expressive power and rotational equivariance. We show that this approach achieves a 7x-30x speedup compared to conventional $\mathrm{SO}(3)$ convolutions. Furthermore, our empirical results demonstrate that the derived E2Former mitigates the computational challenges of existing approaches without compromising the ability to capture detailed geometric information. This development could suggest a promising direction for scalable and efficient molecular modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑