一种面向文本属性图的纯Transformer预训练框架
A Pure Transformer Pretraining Framework on Text-attributed Graphs
AI总结:
针对图领域预训练受特征与结构异质性限制的问题,提出以特征为中心的GSPT框架,用Transformer结合随机游走与掩码特征重构学习图交互模式,在同域图间迁移性更优,适配节点分类与链路预测任务。
摘要:
预训练在从大规模数据中获取通用知识方面发挥着关键作用,CV和NLP领域的大模型已证明其取得了显著成功。然而,由于特征异质性、结构异质性等根本性挑战,图领域的相关进展仍然有限。\n近年来,越来越多的研究尝试在文本属性图(text-attributed graphs, TAGs)上利用大语言模型(Large Language Models, LLMs)提升节点特征质量,其效果优于传统的词袋或word2vec技术。这些高质量的节点特征削弱了此前图结构的核心作用,使得图神经网络(Graph Neural Networks, GNNs)与不依赖结构的多层感知机(Multi-Layer Perceptrons, MLPs)之间的性能差距较小。\n受此启发,我们提出了一种以特征为中心的预训练视角:将图结构作为先验信息,利用丰富统一的特征空间学习可在不同图之间泛化的精细交互模式。我们的框架——基于Transformer的图序列预训练(Graph Sequence Pretraining with Transformer, GSPT)通过随机游走采样节点上下文,采用掩码特征重构任务,使用标准Transformer捕捉LLM统一特征空间中的节点对邻近性。\n该框架利用统一的文本表示而非各异的图结构,在同领域的不同图之间实现了显著更优的迁移能力。GSPT可轻松适配节点分类和链路预测任务,在多个数据集上展现出良好的实验效果。
英文摘要:
Pretraining plays a pivotal role in acquiring generalized knowledge from large-scale data, achieving remarkable successes as evidenced by large models in CV and NLP. However, progress in the graph domain remains limited due to fundamental challenges such as feature heterogeneity and structural heterogeneity. Recently, increasing efforts have been made to enhance node feature quality with Large Language Models (LLMs) on text-attributed graphs (TAGs), demonstrating superiority to traditional bag-of-words or word2vec techniques. These high-quality node features reduce the previously critical role of graph structure, resulting in a modest performance gap between Graph Neural Networks (GNNs) and structure-agnostic Multi-Layer Perceptrons (MLPs). Motivated by this, we introduce a feature-centric pretraining perspective by treating graph structure as a prior and leveraging the rich, unified feature space to learn refined interaction patterns that generalizes across graphs. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks and employs masked feature reconstruction to capture pairwise proximity in the LLM-unified feature space using a standard Transformer. By utilizing unified text representations rather than varying structures, our framework achieves significantly better transferability among graphs within the same domain. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。