面向异构且长尾数据的CLIP引导联邦学习
CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data
AI总结：
针对联邦学习中用户异构与类别分布不平衡的挑战，提出CLIP引导的CLIP2FL方法，通过客户端知识蒸馏和服务器端原型对比学习生成联邦特征重训练分类器，提升异构长尾数据下的联邦学习效果。
摘要：
联邦学习（FL）提供了一种去中心化的机器学习范式，其中服务器与一组客户端协作训练全局模型，且无需访问客户端的数据。用户异构性是联邦学习面临的一项重大挑战，它与类别分布不平衡问题叠加，进一步加大了联邦学习的难度。大型视觉语言模型已取得重大进展，例如对比语言-图像预训练（Contrastive Language-Image Pre-training, CLIP），为图像分类和目标识别开辟了新路径。受CLIP在少样本和零样本学习领域取得的成功启发，我们利用CLIP在其视觉-语言监督下优化服务器与客户端模型之间的联邦学习过程。得益于CLIP强大的跨模态表征能力和丰富的开放词汇先验知识，该方法有望缓解用户异构性和类别分布不平衡问题。本文提出了面向异构且长尾数据的CLIP引导联邦学习方法（CLIP-guided FL, CLIP2FL）。在CLIP2FL中，我们将现成CLIP模型的知识迁移至客户端-服务器模型，并在客户端与服务器之间搭建桥梁。具体而言，在客户端学习阶段，我们在客户端模型与CLIP之间开展知识蒸馏，以提升客户端的特征表征能力。在服务器端学习阶段，为缓解异构性和类别分布不平衡问题，我们生成联邦特征以重新训练服务器模型。我们引入了在CLIP文本编码器监督下的原型对比学习，依据客户端梯度生成联邦特征，并利用这些特征重新训练一个平衡的服务器分类器。
英文摘要：
Federated learning (FL) provides a decentralized machine learning paradigm where a server collaborates with a group of clients to learn a global model without accessing the clients' data. User heterogeneity is a significant challenge for FL, which together with the class-distribution imbalance further enhances the difficulty of FL. Great progress has been made in large vision-language models, such as Contrastive Language-Image Pre-training (CLIP), which paves a new way for image classification and object recognition. Inspired by the success of CLIP on few-shot and zero-shot learning, we use CLIP to optimize the federated learning between server and client models under its vision-language supervision. It is promising to mitigate the user heterogeneity and class-distribution balance due to the powerful cross-modality representation and rich open-vocabulary prior knowledge. In this paper, we propose the CLIP-guided FL (CLIP2FL) method on heterogeneous and long-tailed data. In CLIP2FL, the knowledge of the off-the-shelf CLIP model is transferred to the client-server models, and a bridge is built between the client and server. Specifically, for client-side learning, knowledge distillation is conducted between client models and CLIP to improve the ability of client-side feature representation. For server-side learning, in order to mitigate the heterogeneity and class-distribution imbalance, we generate federated features to retrain the server model. A prototype contrastive learning with the supervision of the text encoder of CLIP is introduced to generate federated features depending on the client-side gradients, and they are used to retrain a balanced server classifier.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
