SwinGNN:重新思考图生成扩散模型中的置换不变性
SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generation
AI总结:
本研究分析了置换不变图生成扩散模型的学习难点,提出非不变模型SwinGNN,结合边到边2-WL消息传递与移位窗口自注意力,辅以关键训练采样技术及随机置换后处理,在多数据集上取得SOTA性能。
摘要:
基于置换等变网络的扩散模型能够学习图数据的置换不变分布。但我们发现,与非不变的同类模型相比,这些不变模型面临更严峻的学习挑战,原因有二:其一,它们的有效目标分布呈现出更多模态;其二,它们的最优单步去噪得分是分量更多的高斯混合模型的得分函数。受此分析启发,我们提出一种非不变扩散模型,名为SwinGNN,该模型采用高效的边到边2-WL消息传递网络,并借鉴SwinTransformers的思路使用基于移位窗口的自注意力机制。此外,通过系统的消融实验,我们确定了若干关键的训练与采样技术,可显著提升图生成的样本质量。最后,我们提出一种简单的后处理技巧,即对生成的图进行随机置换,该方法可证明能将任意图生成模型转换为置换不变模型。在合成数据集以及真实的蛋白质、分子数据集上开展的大量实验表明,我们的SwinGNN取得了当前最优性能。代码已发布于https://github.com/qiyan98/SwinGNN。
英文摘要:
Diffusion models based on permutation-equivariant networks can learn permutation-invariant distributions for graph data. However, in comparison to their non-invariant counterparts, we have found that these invariant models encounter greater learning challenges since 1) their effective target distributions exhibit more modes; 2) their optimal one-step denoising scores are the score functions of Gaussian mixtures with more components. Motivated by this analysis, we propose a non-invariant diffusion model, called $\textit{SwinGNN}$, which employs an efficient edge-to-edge 2-WL message passing network and utilizes shifted window based self-attention inspired by SwinTransformers. Further, through systematic ablations, we identify several critical training and sampling techniques that significantly improve the sample quality of graph generation. At last, we introduce a simple post-processing trick, $\textit{i.e.}$, randomly permuting the generated graphs, which provably converts any graph generative model to a permutation-invariant one. Extensive experiments on synthetic and real-world protein and molecule datasets show that our SwinGNN achieves state-of-the-art performances. Our code is released at https://github.com/qiyan98/SwinGNN.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。