Federated Learning with Label-Masking Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, UCAS（中国科学院大学网络空间安全学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）
Federated learning provides a privacy-preserving manner to collaboratively train models on data distributed over multiple local clients via the coordination of a global server. In this paper, we focus on label distribution skew in federated learning, where due to the different user behavior of the client, label distributions between different clients are significantly different. When faced with such cases, most existing methods will lead to a suboptimal optimization due to the inadequate utilization of label distribution information in clients. Inspired by this, we propose a label-masking distillation approach termed FedLMD to facilitate federated learning via perceiving the various label distributions of each client. We classify the labels into majority and minority labels based on the number of examples per class during training. The client model learns the knowledge of majority labels from local data. The process of distillation masks out the predictions of majority labels from the global model, so that it can focus more on preserving the minority label knowledge of the client. A series of experiments show that the proposed approach can achieve state-of-the-art performance in various cases. Moreover, considering the limited resources of the clients, we propose a variant FedLMD-Tf that does not require an additional teacher, which outperforms previous lightweight approaches without increasing computational costs. Our code is available at https://github.com/wnma3mz/FedLMD.
Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Laboratory（中关村实验室）；Beijing Jiaotong University（北京交通大学）
针对联邦蒸馏的通信开销问题，本文提出 ALU 方法，通过累积多轮本地更新后再向中央服务器传输知识，降低通信频率并显著提升通信效率。
Personalized Federated Learning via Backbone Self-Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, UCAS（中国科学院大学网络空间安全学院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）
Towards Personalized Federated Learning via Comprehensive Knowledge Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
FedUKD: Federated UNet Model with Knowledge Distillation for Land Use Classification from Satellite and Street Views
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Vellore Institute of Technology（韦洛尔理工学院）；College of Engineering, Anna University（安娜大学工程学院）；Measi Academy Of Architecture（米西建筑学院）
Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare
Indian Institute of Technology, Bombay（印度理工学院孟买分校）；Sardar Patel Institute of Technology（萨达尔·帕特尔理工学院）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）；State University of New York at Buffalo（纽约州立大学布法罗分校）
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；International School, Beijing University of Posts and Telecommunications（北京邮电大学国际学院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
Spectral Co-Distillation for Personalized Federated Learning
Singapore University of Technology and Design (SUTD)（新加坡科技设计大学）；Zhejiang University/University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟校区联合学院，浙江大学）
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Faculty of Applied Sciences, Macao Polytechnic University（澳门理工大学应用科学学院）；School of Engineering and Technology, Central Queensland University（中央昆士兰大学工程与技术学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
Federated Learning via Input-Output Collaborative Distillation
University at Buffalo（布法罗大学）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Harvard Medical School（哈佛医学院）；Virginia Tech（弗吉尼亚理工大学）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室杰瑞斯研究中心）；United Imaging Intelligence（联影智能）；Hangzhou Research Institute, Beihang University（北京航空航天大学杭州研究院）；Zhongguancun Laboratory（中关村实验室）；Nanchang Institute of Technology（南昌工学院）
