arXiv 2504.210632025-04-29cs.LGcs.AI

基于无参数路由的Token级提示混合用于联邦领域泛化

Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

School of Computing and Artificial Intelligence, Shandong University of Finance and Economics（计算机与人工智能学院，山东财经大学）；School of Computer Science and Technology, Shandong Jianzhu University（计算机科学与技术学院，山东建筑大学）；College of Electronics and Information Engineering, Tongji University（电子信息工程学院，同济大学）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）

Shuai Gong, Chaoran Cui, Xiaolin Dong, Xiushan Nie, Lei Zhu, Xiaojun Chang

备注：: The manuscript has been submitted to IEEE Transactions on Knowledge and Data Engineering

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对联邦领域泛化中单一全局提示导致个性化样本性能下降及现有MoE方法通信成本高的问题，提出基于无参数路由的Token级提示混合框架TRIP，通过token聚类和最优传输实现细粒度专家分配，在仅需1K参数通信的条件下取得最优泛化效果。

摘要：

联邦领域泛化（FedDG）旨在从具有异构数据的去中心化客户端中学习一个全局可泛化的模型，同时保护隐私。近期的研究通过学习单一全局提示，引入提示学习来适应FedDG中的视觉-语言模型（VLMs）。然而，这种单一提示适用所有的学习范式通常会导致在个性化样本上的性能下降。尽管专家混合（MoE）为专业化提供了有前景的解决方案，但现有的基于MoE的方法存在粗略的图像级专家分配问题，以及参数化路由器带来的高通信成本。为了解决这些局限性，我们提出了TRIP，一个用于FedDG的Token级提示混合与无参数路由框架，将多个提示视为不同的专家。与现有的图像级路由设计不同，TRIP将图像内的不同token分配给特定的专家。为确保通信效率，TRIP结合了基于token聚类和最优传输的无参数路由机制。然后通过聚合专家来合成实例特定的提示，其权重由分配给每个专家的token数量决定。此外，TRIP利用VLM的零样本泛化能力，开发了一种用于提示专家的无偏学习策略。在四个基准上的广泛实验表明，TRIP实现了最优的泛化结果，每轮通信仅需1K参数。我们的代码可在https://github.com/GongShuai8210/TRIP获取。

英文摘要：

Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data while preserving privacy. Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt. However, such a one-prompt-fits-all learning paradigm typically leads to performance degradation on personalized samples. Although the mixture of experts (MoE) offers a promising solution for specialization, existing MoE-based methods suffer from coarse image-level expert assignment and high communication costs from parameterized routers. To address these limitations, we propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG, which treats multiple prompts as distinct experts. Unlike existing image-level routing designs, TRIP assigns different tokens within an image to specific experts. To ensure communication efficiency, TRIP incorporates a parameter-free routing mechanism based on token clustering and optimal transport. The instance-specific prompt is then synthesized by aggregating experts, weighted by the number of tokens assigned to each. Additionally, TRIP develops an unbiased learning strategy for prompt experts, leveraging the VLM's zero-shot generalization capability. Extensive experiments across four benchmarks demonstrate that TRIP achieves optimal generalization results, with communication of only 1K parameters per round. Our code is available at https://github.com/GongShuai8210/TRIP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。