AI 中文总结
针对联邦LoRA中共享结构脆弱的问题，本文提出输入感知动作矩阵揭示稳健共享子空间，并据此设计FedSAIL方法，通过子空间正则化提升性能并大幅降低通信成本。
AI 中文摘要
联邦低秩适应（LoRA）需要确定一个在异构客户端之间共享的更新结构。先前的研究报告称，不同客户端训练得到的LoRA投影矩阵之间存在很强的相似性；然而，这种一致性可能很大程度上是由共同的初始化引起的，并且在独立初始化下会退化为随机重叠。更为关键的是，仅依赖参数相似性本质上忽略了局部输入分布的影响。为了揭示更稳健的共享结构，我们引入了一种输入感知的动作矩阵，该矩阵通过局部层输入的二阶矩统计量来加权适配器更新。实验表明，当参数相似性消失时，该动作矩阵的主要右奇异方向在客户端之间仍然保持强对齐。这种共享的几何结构保留了任务条件差异，并自然地随网络深度而变化。受这些发现的启发，我们提出了联邦子空间引导的动作信息学习（FedSAIL）。FedSAIL不是对权重进行平均，而是估计一个共享的动作子空间来正则化局部训练，同时保留客户端特定的系数。在多个基准测试中，我们的方法在显著降低通信成本的同时， consistently 优于竞争性的联邦LoRA方法，提升了预测性能。
英文摘要
Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.
Comments26 pages, 6 figures, 18 tables
