GNNMerge:无需访问训练数据的GNN模型合并
GNNMerge: Merging of GNN Models Without Accessing Training Data
AI总结:
针对GNN模型合并中共享初始化假设不适用的问题,提出任务无关节点嵌入对齐策略GNNMerge,实现无需训练数据的模型合并,准确率提升达24%,速度提升超两个数量级。
摘要:
模型合并在机器学习中已崭露头角,成为一种无需访问原始训练数据即可将多个训练好的模型整合为单一模型的方法。尽管现有方法已在计算机视觉和自然语言处理等领域取得成功,但其在图神经网络(GNNs)上的应用仍未得到探索。这些方法往往依赖于共享初始化的假设,而这很少适用于GNN。在本工作中,我们首次对GNN的模型合并算法进行了基准研究,揭示了它们在此情境下的有限有效性。为应对这些挑战,我们提出了GNNMerge,它利用一种任务无关的节点嵌入对齐策略来合并GNN。此外,我们证明在温和的松弛条件下,所提出的优化目标对于广泛使用的GNN架构允许直接解析解,显著提升了其计算效率。跨多种数据集、任务和架构的实证评估表明,GNNMerge相比现有方法准确率最高提升24%,同时相比从头训练实现了超过两个数量级的加速。
英文摘要:
Model merging has gained prominence in machine learning as a method to integrate multiple trained models into a single model without accessing the original training data. While existing approaches have demonstrated success in domains such as computer vision and NLP, their application to Graph Neural Networks (GNNs) remains unexplored. These methods often rely on the assumption of shared initialization, which is seldom applicable to GNNs. In this work, we undertake the first benchmarking study of model merging algorithms for GNNs, revealing their limited effectiveness in this context. To address these challenges, we propose GNNMerge, which utilizes a task-agnostic node embedding alignment strategy to merge GNNs. Furthermore, we establish that under a mild relaxation, the proposed optimization objective admits direct analytical solutions for widely used GNN architectures, significantly enhancing its computational efficiency. Empirical evaluations across diverse datasets, tasks, and architectures establish GNNMerge to be up to 24% more accurate than existing methods while delivering over 2 orders of magnitude speed-up compared to training from scratch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。