GNNs, like other deep learning models, are data and computation hungry. There is a pressing need to scale training of GNNs on large datasets to enable their usage on low-resource environments. Graph distillation is an effort in that direction with the aim to construct a smaller synthetic training set from the original training data without significantly compromising model performance. While initial efforts are promising, this work is motivated by two key observations: (1) Existing graph distillation algorithms themselves rely on training with the full dataset, which undermines the very premise of graph distillation. (2) The distillation process is specific to the target GNN architecture and hyper-parameters and thus not robust to changes in the modeling pipeline. We circumvent these limitations by designing a distillation algorithm called Mirage for graph classification. Mirage is built on the insight that a message-passing GNN decomposes the input graph into a multiset of computation trees. Furthermore, the frequency distribution of computation trees is often skewed in nature, enabling us to condense this data into a concise distilled summary. By compressing the computation data itself, as opposed to emulating gradient flows on the original training set-a prevalent approach to date-Mirage transforms into an unsupervised and architecture-agnostic distillation algorithm. Extensive benchmarking on real-world datasets underscores Mirage's superiority, showcasing enhanced generalization accuracy, data compression, and distillation efficiency when compared to state-of-the-art baselines.
Adversary Agnostic Robust Deep Reinforcement Learning
Nanyang Technological University(南洋理工大学);Singapore Institute of Manufacturing Technology(新加坡制造技术研究院);Agency for Science, Technology and Research(新加坡科技研究局);Macquarie University(麦考瑞大学)
Neural Tangent Knowledge Distillation for Optical Convolutional Networks
Department of Electrical and Computer Engineering, University of Washington(电气与计算机工程系,华盛顿大学);Department of Physics, University of Washington(物理系,华盛顿大学);Department of Applied Mathematics, University of Washington(应用数学系,华盛顿大学)
Scalable Model Merging with Progressive Layer-wise Distillation
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);School of Computer Science, Beijing Institute of Technology(北京理工大学计算机学院);Shanghai Qizhi Institute(上海启智研究院)
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
University of British Columbia(不列颠哥伦比亚大学);University of Helsinki(赫尔辛基大学);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)