arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.14845cs.LG
收藏

ULTRA-DP:通过多任务图双提示统一图预训练

ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt

Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);Salesforce Research Asia(Salesforce亚洲研究院);University of Virginia(弗吉尼亚大学)

Mouxiang Chen, Zemin Liu, Chenghao Liu, Jundong Li, Qiheng Mao, Jianling Sun

AI总结:

针对图预训练的语义鸿沟与位置感知不足问题,提出融合任务与位置提示的ULTRA-DP统一框架,搭配多尺度前置任务与可迁移性测试,显著提升预训练性能与通用性。

摘要:

近期研究已证明预训练图神经网络(GNNs)在捕获可迁移图语义、提升各类下游任务性能方面的有效性。然而,从前置任务中学到的语义知识可能与下游任务无关,导致语义鸿沟,限制了图预训练的应用。为缩小这一鸿沟,传统方法提出混合预训练,以多任务学习的方式将各类前置任务结合起来,学习多粒度知识,但这类方法无法区分不同任务,会导致部分可迁移的任务特定知识相互扭曲。此外,大多数GNN无法区分位于图中不同位置的节点,使其无法学习位置特定知识,进而导致性能次优。在本研究中,受自然语言处理中基于提示的调优启发,我们提出了一个图混合预训练统一框架,通过提示机制将任务识别与位置识别注入GNN,即多任务图双提示(ULTRA-DP)。基于该框架,我们提出一种基于提示的可迁移性测试,用于找到最相关的前置任务,从而缩小语义鸿沟。为实现混合预训练任务,除经典的边预测任务(节点-节点层级)外,我们进一步提出了一种基于一组k近邻(节点-组层级)的新型预训练范式。不同尺度的任务相结合能够全面表达更多结构语义,衍生出更丰富的多粒度知识。大量实验表明,我们提出的ULTRA-DP能够显著提升混合预训练方法的性能,且对其他预训练任务和骨干架构具有通用性。

英文摘要:

Recent research has demonstrated the efficacy of pre-training graph neural networks (GNNs) to capture the transferable graph semantics and enhance the performance of various downstream tasks. However, the semantic knowledge learned from pretext tasks might be unrelated to the downstream task, leading to a semantic gap that limits the application of graph pre-training. To reduce this gap, traditional approaches propose hybrid pre-training to combine various pretext tasks together in a multi-task learning fashion and learn multi-grained knowledge, which, however, cannot distinguish tasks and results in some transferable task-specific knowledge distortion by each other. Moreover, most GNNs cannot distinguish nodes located in different parts of the graph, making them fail to learn position-specific knowledge and lead to suboptimal performance. In this work, inspired by the prompt-based tuning in natural language processing, we propose a unified framework for graph hybrid pre-training which injects the task identification and position identification into GNNs through a prompt mechanism, namely multi-task graph dual prompt (ULTRA-DP). Based on this framework, we propose a prompt-based transferability test to find the most relevant pretext task in order to reduce the semantic gap. To implement the hybrid pre-training tasks, beyond the classical edge prediction task (node-node level), we further propose a novel pre-training paradigm based on a group of $k$-nearest neighbors (node-group level). The combination of them across different scales is able to comprehensively express more structural semantics and derive richer multi-grained knowledge. Extensive experiments show that our proposed ULTRA-DP can significantly enhance the performance of hybrid pre-training methods and show the generalizability to other pre-training tasks and backbone architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑