发表机构
Beijing Institute of Technology; Shandong University; Institute of Information Engineering; University of Science and Technology Beijing（北京理工大学; 山东大学; 信息工程研究所; 北京科技大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对联邦多模态图基础模型中编码器与GNN适配不兼容的问题，提出FedCORE框架，通过共享低维潜在状态联合优化感知与推理，将配对差距从30.6降至5.9，降幅80.7%。
AI 中文摘要
联邦多模态图基础模型（GFMs）旨在将预训练的多模态模型适配到分散的图数据上，其中每个客户端拥有一个私有多模态图，且不能共享原始信息。这些模型通常结合一个从异构模态中提取语义证据的多模态编码器和一个在图结构上进行关系推理的图神经网络（GNN）。然而，现有的联邦GFM适配方法主要更新图侧模块，同时保持多模态编码器冻结，从而将适配限制在“信息如何传播”上，而固定了“提取什么信息”。通过实证研究，我们揭示了编码器和GNN的适配并非独立：编码器适配受到图关系的影响，而跨客户端的模块交换揭示了分别参数化的编码器和GNN更新之间存在显著的配对敏感性。受此观察启发，我们提出了FedCORE，一种联邦适配框架，通过共享的低维潜在状态来表示编码器和GNN的更新。FedCORE从多模态和结构信号中联合优化这一核心，并直接在共享状态空间中进行联邦演化，从而保持感知和推理适配之间的兼容性。大量实验表明，与独立的联合适配相比，FedCORE将编码器-GNN配对差距从30.6降至5.9，对应80.7%的降低。
英文摘要
Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to \emph{how information is propagated} while fixing \emph{what information is extracted}. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose \textbf{FedCORE}, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder--GNN pairing gap from $30.6$ to $5.9$, corresponding to an $80.7\%$ reduction over independent joint adaptation.
