基于视觉-语言基础模型的联邦学习：理论分析与方法
Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
AI总结：
本文通过特征学习理论构建基于提示的联邦学习理论分析框架，提出全局提示与局部提示组合方法以平衡泛化与个性化，并推导出最优混合系数。
摘要：
将CLIP等预训练视觉-语言基础模型集成到联邦学习中，因能增强跨多样化任务的泛化能力而受到广泛关注。通常，视觉-语言模型的联邦学习采用提示学习来降低通信和计算成本，即基于提示的联邦学习。然而，目前对基于提示的联邦学习性能的理解缺乏理论分析。在本工作中，我们通过特征学习理论构建了基于提示的联邦学习的理论分析框架。具体而言，我们监测基于提示的联邦学习中信号学习和噪声记忆的演化过程，证明性能可以通过任务相关与任务无关系数的比率来评估。此外，我们将投资组合优化中的收益与风险类比于特征学习中的任务相关项与任务无关项。借鉴投资组合优化中组合两个独立资产可在降低风险的同时保持收益的思想，我们引入两个提示：全局提示和局部提示，构建一个提示组合以平衡泛化与个性化。由此，我们展示了提示组合的性能优势，并推导出最优混合系数。这些理论论断得到了实证实验的进一步支持。
英文摘要：
Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
