arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.09474cs.LGcs.AI
收藏

DIFFormer:由能量约束扩散诱导的可扩展(图)Transformer

DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion

Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);University of Oxford(牛津大学);Amazon Web Service(亚马逊云科技)

Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, Junchi Yan

AI总结:

本文提出 DIFFormer,通过能量约束扩散过程建模实例间依赖并诱导可扩展 Transformer 编码器,在大规模图节点分类、半监督分类和时空预测等任务中取得优越性能。

摘要:

现实世界的数据生成通常涉及实例之间复杂的相互依赖关系,这违反了标准学习范式的独立同分布假设,并对揭示用于学习所需实例表示的几何结构提出了挑战。为此,我们引入了一种能量约束扩散模型,该模型将数据集中的一批实例编码为演化状态,通过实例间的交互逐步融入其他实例的信息。扩散过程受到关于一个原则性能量函数的下降准则约束,该能量函数刻画了实例表示在潜在结构上的全局一致性。我们提供了严格的理论,推导出任意实例对之间成对扩散强度的闭式最优估计,从而产生了一类新的神经编码器,称为 DIFFormer(基于扩散的 Transformer),并具有两种实例化形式:一种具有线性复杂度的简单版本,适用于实例数量极大的情况;另一种用于学习复杂结构的高级版本。实验凸显了我们的模型作为通用编码器骨干的广泛适用性,在多种任务中表现出优越性能,例如大规模图上的节点分类、半监督图像/文本分类以及时空动态预测。

英文摘要:

Real-world data generation often involves complex inter-dependencies among instances, violating the IID-data hypothesis of standard learning paradigms and posing a challenge for uncovering the geometric structures for learning desired instance representations. To this end, we introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states that progressively incorporate other instances' information by their interactions. The diffusion process is constrained by descent criteria w.r.t.~a principled energy function that characterizes the global consistency of instance representations over latent structures. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs, which gives rise to a new class of neural encoders, dubbed as DIFFormer (diffusion-based Transformers), with two instantiations: a simple version with linear complexity for prohibitive instance numbers, and an advanced version for learning complex structures. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks, such as node classification on large graphs, semi-supervised image/text classification, and spatial-temporal dynamics prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑