pFedMMA: 为视觉-语言模型设计的个性化联邦微调多模态适配器
pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
AI总结：
pFedMMA通过多模态适配器实现视觉-语言模型的个性化联邦微调，平衡个性化与泛化能力，优于现有方法。
摘要：
视觉-语言模型（VLMs）如CLIP在零样本和少样本设置中表现出色，但将其高效适应到分散、异构的数据仍是一个挑战。尽管提示微调已成为个性化联邦学习中流行的参数高效方法，现有方法往往牺牲泛化能力以换取个性化，特别是在面对未见类别或领域时表现不佳。在本工作中，我们提出pFedMMA，这是首个利用多模态适配器进行视觉-语言任务的个性化联邦学习框架。每个适配器包含模态特定的上/下投影层以及一个全局共享的投影层，用于对齐跨模态特征。我们的优化策略允许客户端本地适应个性化数据分布，同时协同训练共享投影以提高全局泛化能力。这种设计也是高效的，因为通信轮次中仅交换共享组件。通过在十一组数据集上的广泛实验，包括领域和标签偏移场景，我们证明pFedMMA在个性化与泛化之间的最佳权衡，优于最近的联邦提示微调方法。
英文摘要：
Vision-Language Models (VLMs) like CLIP have demonstrated remarkable generalization in zero- and few-shot settings, but adapting them efficiently to decentralized, heterogeneous data remains a challenge. While prompt tuning has emerged as a popular parameter-efficient approach in personalized federated learning, existing methods often sacrifice generalization in favor of personalization, struggling particularly on unseen classes or domains. In this work, we propose pFedMMA, the first personalized federated learning framework that leverages multi-modal adapters for vision-language tasks. Each adapter contains modality-specific up- and down-projection layers alongside a globally shared projection that aligns cross-modal features. Our optimization strategy allows clients to locally adapt to personalized data distributions while collaboratively training the shared projection to improve global generalization. This design is also communication-efficient, as only the shared component is exchanged during communication rounds. Through extensive experiments across eleven datasets, including domain- and label-shift scenarios, we show that pFedMMA achieves state-of-the-art trade-offs between personalization and generalization, outperforming recent federated prompt tuning methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
