arXiv 2409.060672026-07-08cs.AIcs.CLcs.LG

MLLM-LLaVA-FL：多模态大语言模型辅助联邦学习

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Duke University（杜克大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Maryland College Park（马里兰大学学院市分校）；Lenovo Research（联想研究院）

Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, Hai Li

备注：: Accepted to WACV 2025
发表信息：: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对联邦学习中数据异质性问题，提出MLLM-LLaVA-FL框架，利用多模态大语言模型，通过全球视觉文本预训练、客户端本地训练和服务器端全局对齐三个阶段，提升联邦学习性能。

摘要：

以往联邦学习（FL）研究常因不同客户端数据异质性而性能下降。鉴于多模态大语言模型（MLLMs）如GPT-4v和LLaVA在多模态任务中表现出色，本文引入多模态大语言模型辅助联邦学习（MLLM-LLaVA-FL）框架。该框架利用MLLMs的跨模态表示能力和开源数据，在服务器端解决异质性和长尾挑战。其有三个关键阶段：全球视觉文本预训练、客户端本地训练、服务器端全局对齐。实验表明，该框架在FL数据异质性和长尾分布场景中性能良好。

英文摘要：

Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs), such as GPT-4v and LLaVA, which demonstrate their exceptional proficiency in multimodal tasks, such as image captioning and multimodal question answering. We introduce a novel federated learning framework, named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL), which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs, our framework is adept at harnessing the extensive, yet previously underexploited, open-source data accessible from websites and powerful server-side computational resources. Hence, the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices, distinguishing it from prior methodologies. Our framework has three key stages. Initially, we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online, with the assistance of MLLMs. Subsequently, the pretrained model is distributed among various clients for local training. Finally, once the locally trained models are transmitted back to the server, a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks, show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。