Knowledge distillation (KD) has shown to be effective to boost the performance of graph neural networks (GNNs), where the typical objective is to distill knowledge from a deeper teacher GNN into a shallower student GNN. However, it is often quite challenging to train a satisfactory deeper GNN due to the well-known over-parametrized and over-smoothing issues, leading to invalid knowledge transfer in practical applications. In this paper, we propose the first Free-direction Knowledge Distillation framework via reinforcement learning for GNNs, called FreeKD, which is no longer required to provide a deeper well-optimized teacher GNN. Our core idea is to collaboratively learn two shallower GNNs to exchange knowledge between them. As we observe that one typical GNN model often exhibits better and worse performances at different nodes during training, we devise a dynamic and free-direction knowledge transfer strategy that involves two levels of actions: 1) node-level action determines the directions of knowledge transfer between the corresponding nodes of two networks; and then 2) structure-level action determines which of the local structures generated by the node-level actions to be propagated. Additionally, considering that different augmented graphs can potentially capture distinct perspectives of the graph data, we propose FreeKD-Prompt that learns undistorted and diverse augmentations based on prompt learning for exchanging varied knowledge. Furthermore, instead of confining knowledge exchange within two GNNs, we develop FreeKD++ to enable free-direction knowledge transfer among multiple GNNs. Extensive experiments on five benchmark datasets demonstrate our approaches outperform the base GNNs in a large margin. More surprisingly, our FreeKD has comparable or even better performance than traditional KD algorithms that distill knowledge from a deeper and stronger teacher GNN.
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs
University of Minnesota（明尼苏达大学）；Amazon Web Services（亚马逊云科技）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）
提出 GRAD 图感知蒸馏框架，通过共享 LM 联合训练 GNN 教师与无图学生，实现快速无图推理并提升节点分类性能。
FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition
The Hong Kong University of Science and Technology（香港科技大学）；University of Girona（赫罗纳大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（港科大深港协同创新研究院）
Generalist Graph Anomaly Detection via Prototype-Based Distillation
School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an, China（西安交通大学计算机科学与技术学院）；National Engineering Research Center for Visual Information and Applications, Xi'an, China（视觉信息与应用国家工程研究中心）；University of Virginia, Charlottesville, USA（弗吉尼亚大学）；University of Central Florida, Orlando, USA（佛罗里达大学）；School of Distance Education, Xi’an Jiaotong University, Xi'an, China（西安交通大学继续教育学院）；School of Cyber Science and Engineering, Xi'an Jiaotong University, Xi'an, China（西安交通大学网络安全学院）
Distilling A Universal Expert from Clustered Federated Learning
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education（教育部符号计算与知识工程重点实验室）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney（悉尼大学技术学院澳大利亚人工智能研究所）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机科学学院）
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Faculty of Applied Sciences, Macao Polytechnic University（澳门理工大学应用科学学院）；School of Engineering and Technology, Central Queensland University（中央昆士兰大学工程与技术学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
