采用异构压缩客户端的联邦基础模型微调

中文摘要 AI 辅助

基础模型的联邦学习面临一个根本性的资源不对称挑战：拥有最具价值的特定领域数据的机构无法承载数十亿参数的模型。现有的异构联邦方法尝试通过参数高效微调、模型剪枝或知识蒸馏来弥合这一差距，但每种方法都牺牲了关键特性，无论是全模型内存缩减、架构自包含性还是表示保真度，都未能解决核心矛盾。我们提出FedSLM，一种以参数为中心的异构压缩客户端联邦微调框架。FedSLM使用基于SVD的分解生成自包含的客户端模型，其低秩子空间形成可聚合的结构兼容嵌套流形。它随后应用两阶段协议：在压缩组内同步轻量级适配器，通过结构对齐跨组融合满秩重构。最后，带有辅助置信度损失的弱到强引出步骤将聚合知识转移到全规模服务器，同时显式偏差-方差权衡缓解压缩伪影。我们为适配器级聚合提供理论保证，为跨组融合提供子空间对齐边界，并表征置信度损失如何缓解弱监督噪声。在自然语言和视觉-语言基准上的实验表明，FedSLM在IID和非IID划分下均优于现有联邦基线，同时客户端模型运行所需的GPU内存约为全模型的50%。

英文摘要

Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

发表机构

• The Hong Kong Polytechnic University（香港理工大学）
• OceanBase, Ant Group（蚂蚁集团OceanBase）
• School of Management, Xi’an Jiaotong University（西安交通大学管理学院）
• School of Mathematics and Statistics, Xidian University（西安电子科技大学数学与统计学院）
• School of Computer Science, Wuhan University（武汉大学计算机学院）
• School of Computer Science, China University of Geosciences（中国地质大学计算机学院）

机构由 AI 辅助整理，请以论文原文为准。