基于多模态原型的开放词汇联邦学习
Open-Vocabulary Federated Learning with Multimodal Prototyping
AI总结：
针对联邦学习中新用户提出任意未见类别查询的开放词汇难题，本文提出Fed-MP框架，利用轻量客户端残差自适应聚合VLM并结合多模态原型机制，提升模型对未知类别的泛化能力。
摘要：
现有的联邦学习（FL）研究通常假设训练标签空间与测试标签空间完全相同。然而，在真实应用中，这一假设过于理想化而难以成立。新用户可能提出涉及未见类别数据的查询，而这类开放词汇查询会直接导致此类FL系统失效。因此，本工作明确聚焦于FL中尚未得到充分研究的开放词汇挑战。也就是说，对于新用户，全局服务器应当理解其涉及任意未知类别的查询。为解决该问题，我们利用预训练视觉-语言模型（VLM）。具体而言，我们提出了一种在FL场景下为VLM量身定制的新型适配框架，名为Federated Multimodal Prototyping（Fed-MP，联邦多模态原型）。Fed-MP基于轻量级客户端残差自适应地聚合本地模型权重，并通过一种新颖的多模态原型机制进行预测。Fed-MP利用从已见类别中学到的知识，增强适配后VLM对未见类别的鲁棒性。我们在多个数据集上的实证评估验证了Fed-MP的有效性。
英文摘要：
Existing federated learning (FL) studies usually assume the training label space and test label space are identical. However, in real-world applications, this assumption is too ideal to be true. A new user could come up with queries that involve data from unseen classes, and such open-vocabulary queries would directly defect such FL systems. Therefore, in this work, we explicitly focus on the under-explored open-vocabulary challenge in FL. That is, for a new user, the global server shall understand her/his query that involves arbitrary unknown classes. To address this problem, we leverage the pre-trained vision-language models (VLMs). In particular, we present a novel adaptation framework tailored for VLMs in the context of FL, named as Federated Multimodal Prototyping (Fed-MP). Fed-MP adaptively aggregates the local model weights based on light-weight client residuals, and makes predictions based on a novel multimodal prototyping mechanism. Fed-MP exploits the knowledge learned from the seen classes, and robustifies the adapted VLM to unseen categories. Our empirical evaluation on various datasets validates the effectiveness of Fed-MP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
