arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.09486cs.LGcs.AI
收藏

Mirage:面向图分类的模型无关图蒸馏

Mirage: Model-Agnostic Graph Distillation for Graph Classification

Yardi School of Artificial Intelligence(亚迪人工智能学院);Indian Institute of Technology, Delhi(印度理工学院德里分校)

Mridul Gupta, Sahil Manchanda, Hariprasad Kodamana, Sayan Ranu

AI总结:

Mirage提出一种模型无关的无监督图蒸馏方法,通过压缩GNN计算树的多重集频率分布来构建小型合成训练集,从而在保持图分类性能的同时提升数据压缩与蒸馏效率。

摘要:

图神经网络(GNN)与其他深度学习模型一样,对数据和计算资源需求巨大。为了在低资源环境中使用GNN,迫切需要在大规模数据集上扩展GNN的训练。图蒸馏正是朝着这一方向的努力,其目标是在不显著损害模型性能的前提下,从原始训练数据中构建一个更小的合成训练集。尽管初步努力颇具前景,但本工作受到两个关键观察的驱动:(1)现有的图蒸馏算法本身依赖在完整数据集上训练,这削弱了图蒸馏的基本前提。(2)蒸馏过程特定于目标GNN架构和超参数,因此对建模流程的变化不够鲁棒。我们通过设计一种名为Mirage的图分类蒸馏算法来规避这些限制。Mirage建立在这样一个洞察之上:消息传递GNN将输入图分解为计算树的多重集。此外,计算树的频率分布通常具有偏斜特性,使我们能够将这些数据浓缩为简洁的蒸馏摘要。通过压缩计算数据本身,而不是模拟原始训练集上的梯度流——这是迄今为止的主流方法——Mirage转变为一种无监督且与架构无关的蒸馏算法。在真实世界数据集上的广泛基准测试凸显了Mirage的优越性,与最先进的基线相比,展示了更强的泛化精度、数据压缩率和蒸馏效率。

英文摘要:

GNNs, like other deep learning models, are data and computation hungry. There is a pressing need to scale training of GNNs on large datasets to enable their usage on low-resource environments. Graph distillation is an effort in that direction with the aim to construct a smaller synthetic training set from the original training data without significantly compromising model performance. While initial efforts are promising, this work is motivated by two key observations: (1) Existing graph distillation algorithms themselves rely on training with the full dataset, which undermines the very premise of graph distillation. (2) The distillation process is specific to the target GNN architecture and hyper-parameters and thus not robust to changes in the modeling pipeline. We circumvent these limitations by designing a distillation algorithm called Mirage for graph classification. Mirage is built on the insight that a message-passing GNN decomposes the input graph into a multiset of computation trees. Furthermore, the frequency distribution of computation trees is often skewed in nature, enabling us to condense this data into a concise distilled summary. By compressing the computation data itself, as opposed to emulating gradient flows on the original training set-a prevalent approach to date-Mirage transforms into an unsupervised and architecture-agnostic distillation algorithm. Extensive benchmarking on real-world datasets underscores Mirage's superiority, showcasing enhanced generalization accuracy, data compression, and distillation efficiency when compared to state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑