arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.08678cs.LG
收藏

Graph Mamba:迈向基于状态空间模型的图学习

Graph Mamba: Towards Learning on Graphs with State Space Models

Cornell University(康奈尔大学)

Ali Behrouz, Farnoosh Hashemi

AI总结:

针对现有GNN的过度挤压、长程依赖捕捉差及GTs计算成本高等问题,提出基于选择性状态空间模型的Graph Mamba Networks框架,通过四个必需和一个可选步骤设计,在大幅降低计算成本的同时于多种基准数据集上取得优异性能。

摘要:

图神经网络(GNNs)在图表征学习方面展现出了有潜力的前景。大多数GNN定义了一种局部消息传递机制,通过堆叠多层来在图上传播信息。然而,已知这些方法存在两个主要局限性:过度挤压和对长程依赖的捕捉能力差。最近,Graph Transformers(GTs)作为消息传递神经网络(MPNNs)的强大替代方案出现。然而,GTs具有二次方的计算成本,缺乏对图结构的归纳偏置,并且依赖于复杂的位置/结构编码(SE/PE)。在本文中,我们表明虽然Transformers、复杂的消息传递和SE/PE在实践中足以取得良好的性能,但它们都不是必需的。受状态空间模型(SSMs)近期成功(如Mamba)的启发,我们提出了Graph Mamba Networks(GMNs),这是一个基于选择性SSMs的新一类GNN的通用框架。我们讨论并分类了将SSMs适应于图结构数据时面临的新挑战,并提出了设计GMNs所需的四个步骤和一个可选步骤,其中我们选择了(1)邻域Token化,(2)Token排序,(3)双向选择性SSM编码器架构,(4)局部编码,以及非必需的(5)PE和SE。我们进一步为GMNs的能力提供了理论证明。实验表明,尽管计算成本要低得多,GMNs在长程、小规模、大规模和异配基准数据集上均取得了出色的性能。

英文摘要:

Graph Neural Networks (GNNs) have shown promising potential in graph representation learning. The majority of GNNs define a local message-passing mechanism, propagating information over the graph by stacking multiple layers. These methods, however, are known to suffer from two major limitations: over-squashing and poor capturing of long-range dependencies. Recently, Graph Transformers (GTs) emerged as a powerful alternative to Message-Passing Neural Networks (MPNNs). GTs, however, have quadratic computational cost, lack inductive biases on graph structures, and rely on complex Positional/Structural Encodings (SE/PE). In this paper, we show that while Transformers, complex message-passing, and SE/PE are sufficient for good performance in practice, neither is necessary. Motivated by the recent success of State Space Models (SSMs), such as Mamba, we present Graph Mamba Networks (GMNs), a general framework for a new class of GNNs based on selective SSMs. We discuss and categorize the new challenges when adapting SSMs to graph-structured data, and present four required and one optional steps to design GMNs, where we choose (1) Neighborhood Tokenization, (2) Token Ordering, (3) Architecture of Bidirectional Selective SSM Encoder, (4) Local Encoding, and dispensable (5) PE and SE. We further provide theoretical justification for the power of GMNs. Experiments demonstrate that despite much less computational cost, GMNs attain an outstanding performance in long-range, small-scale, large-scale, and heterophilic benchmark datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑