FedPIA——利用Wasserstein重心进行置换与集成适配器以在多模态联邦学习中微调基础模型
FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning
AI总结：
针对多模态联邦学习中微调大型视觉语言模型时面临的数据与任务异质性及资源限制问题，提出FedPIA框架，通过利用Wasserstein重心在服务器和客户端进行适配器的逐层置换与集成，有效融合客户端特定与无关知识，在48个医学数据集的2000多次实验中显著优于现有PEFT-FL基线。
摘要：
大型视觉语言模型通常需要大型文本和图像数据集才能进行有效的微调。然而，由于严格的隐私法规，从各个站点（尤其是在医疗保健领域）收集数据具有挑战性。一种替代方案是在终端用户设备（如医疗诊所）上微调这些模型，而无需将数据发送到服务器。这些本地客户端通常计算能力有限且数据集较小，不足以单独完全微调大型VLM。针对这些场景的一种朴素解决方案是利用参数高效微调（PEFT）策略，并应用联邦学习（FL）算法来组合学习到的适配器权重，从而尊重资源限制和数据隐私。然而，这种方法并未充分利用在多样化数据分布和多样化任务上训练的多个适配器的知识。适配器受到客户端间数据异质性和任务异质性的不利影响，导致次优收敛。为此，我们提出了一种名为FedPIA的新颖框架，该框架通过在服务器端引入本地适配器的置换与集成，并在客户端引入全局适配器的置换与集成，利用Wasserstein重心改进客户端特定知识与客户端无关知识的融合，从而改进了FL和PEFT的朴素组合。这种逐层置换有助于在集成前弥合本地适配器和全局适配器参数空间中的差距。我们利用48个医学图像数据集，在涵盖视觉问答以及基于图像和报告的多标签疾病检测的五种不同医学视觉语言FL任务设置中，进行了超过2000次客户端级实验。我们的实验涉及多样化的客户端设置、十种不同模态和两种VLM主干，证明FedPIA始终优于最先进的PEFT-FL基线。
英文摘要：
Large Vision-Language Models typically require large text and image datasets for effective fine-tuning. However, collecting data from various sites, especially in healthcare, is challenging due to strict privacy regulations. An alternative is to fine-tune these models on end-user devices, such as in medical clinics, without sending data to a server. These local clients typically have limited computing power and small datasets, which are not enough for fully fine-tuning large VLMs on their own. A naive solution to these scenarios is to leverage parameter-efficient fine-tuning (PEFT) strategies and apply federated learning (FL) algorithms to combine the learned adapter weights, thereby respecting the resource limitations and data privacy. However, this approach does not fully leverage the knowledge from multiple adapters trained on diverse data distributions and for diverse tasks. The adapters are adversely impacted by data heterogeneity and task heterogeneity across clients resulting in suboptimal convergence. To this end, we propose a novel framework called FedPIA that improves upon the naive combinations of FL and PEFT by introducing Permutation and Integration of the local Adapters in the server and global Adapters in the clients exploiting Wasserstein barycenters for improved blending of client-specific and client-agnostic knowledge. This layerwise permutation helps to bridge the gap in the parameter space of local and global adapters before integration. We conduct over 2000 client-level experiments utilizing 48 medical image datasets across five different medical vision-language FL task settings encompassing visual question answering as well as image and report-based multi-label disease detection. Our experiments involving diverse client settings, ten different modalities, and two VLM backbones demonstrate that FedPIA consistently outperforms the state-of-the-art PEFT-FL baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
