In this paper, to deal with the heterogeneity in federated learning (FL) systems, a knowledge distillation (KD) driven training framework for FL is proposed, where each user can select its neural network model on demand and distill knowledge from a big teacher model using its own private dataset. To overcome the challenge of train the big teacher model in resource limited user devices, the digital twin (DT) is exploit in the way that the teacher model can be trained at DT located in the server with enough computing resources. Then, during model distillation, each user can update the parameters of its model at either the physical entity or the digital agent. The joint problem of model selection and training offloading and resource allocation for users is formulated as a mixed integer programming (MIP) problem. To solve the problem, Q-learning and optimization are jointly used, where Q-learning selects models for users and determines whether to train locally or on the server, and optimization is used to allocate resources for users based on the output of Q-learning. Simulation results show the proposed DT-assisted KD framework and joint optimization method can significantly improve the average accuracy of users while reducing the total delay.
Peak-Controlled Logits Poisoning Attack in Federated Distillation
School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
FedUKD: Federated UNet Model with Knowledge Distillation for Land Use Classification from Satellite and Street Views
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Vellore Institute of Technology（韦洛尔理工学院）；College of Engineering, Anna University（安娜大学工程学院）；Measi Academy Of Architecture（米西建筑学院）
FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）
Towards Personalized Federated Learning via Comprehensive Knowledge Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
FedKDX: Federated Learning with Negative Knowledge Distillation for Enhanced Healthcare AI Systems
Faculty of EEE, Phenikaa School of Engineering（电子工程系，Phenikaa工程学院）；Phenikaa University（Phenikaa大学）；VinUni-Illinois Smart Health Center（VinUniversity-伊利诺伊智能健康中心）；VinUniversity（Vin大学）；College of Engineering & Computer Science（工程与计算机科学学院）
FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Laboratory（中关村实验室）
Jiangsu University（江苏大学）；Southwest University（西南大学）；The University of Sydney（悉尼大学）；Monash University（莫纳什大学）；Sichuan University（四川大学）；Nanyang Technological University（南洋理工大学）
Knowledge Distillation in Federated Edge Learning: A Survey
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Laboratory（中关村实验室）；Beijing Jiaotong University（北京交通大学）
DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devices
School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦女王玛丽大学电子工程与计算机科学学院）；Department of Computer Science, Faculty of Environment, Science and Economy, University of Exeter（埃克塞特大学环境、科学与经济学院计算机科学系）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Beijing Key Laboratory of Petroleum Data Mining, China University of Petroleum（北京石油数据挖掘重点实验室，中国石油大学）
Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University（北京交通大学轨道交通运行安全协同创新中心）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Federated Knowledge Distillation for Multi-Model Architectures Lithography Hotspot Detection
City University of New York（纽约城市大学）；Sichuan Agricultural University（四川农业大学）；The University of Sydney（悉尼大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of St Andrews（圣安德鲁大学）；Southwest University（西南大学）
Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning
College of Informatics Huazhong Agricultural University（华中农业大学信息学院）；Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology（香港科技大学电子与计算机工程系）；NVIDIA Corporation（NVIDIA公司）
