基于多模态知识协作的鲁棒去中心化联邦蒸馏

中文摘要 AI 辅助

本文提出了一种鲁棒的去中心化联邦蒸馏方法，使具有异构模型的客户端能够通过共享未标记公共数据上的预测进行协作。在所提出的方法中，每个客户端首先在类别预测、边界决策和预测相关性三种模态下评估接收到的预测。然后，它过滤掉不可靠的客户端，为保留的客户端分配基于可靠性的权重，并为每种类型的知识构建一个教师模型。最后，使用从私有数据计算的有监督梯度验证相应的蒸馏梯度。在最终模型更新之前，移除冲突的预测和边界梯度，并抑制冲突的关系梯度。我们通过证明在拜占庭蒸馏下诚实客户端的稳定局部优化来证明所提出方法的收敛性。特别地，我们表明我们的方法确保在跨模态融合后，对蒸馏梯度和个体客户端私有梯度的拜占庭影响均有界，从而在拜占庭蒸馏下实现诚实客户端的稳定局部优化。在CIFAR-10和CIFAR-100上的大量实验表明，所提出的方法在非独立同分布数据和拜占庭攻击下提高了客户端异构模型的预测精度。随着联邦学习在边缘计算和任务导向的无人机协作等去中心化环境中的需求日益增长，我们的方法在客户端暴露于接收者特定的恶意预测拜占庭消息的不可靠现实场景中，具有采用DFL的巨大潜力。

英文摘要

This paper propose a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of federated learning in decentralized environments such as edge computing and mission-oriented UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.

发表机构

• Faculty of Applied Sciences, Macao Polytechnic University（澳门理工大学应用科学学院）
• School of Engineering and Technology, Central Queensland University（中央昆士兰大学工程与技术学院）
• School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）

机构由 AI 辅助整理，请以论文原文为准。