arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.06154cs.AI
收藏

摊销联邦自适应:基于超网络的LoRA用于个性化基础模型

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

Indian Institute of Technology, Bombay(印度理工学院班加罗尔)

Sunny Gupta, Shambhavi Shanker, Amit Sethi

AI总结:

提出HyperLoRA框架,通过超网络驱动的LoRA生成和乘积空间聚合,解决联邦LoRA中的结构聚合偏差和客户端初始化滞后问题,实现高效个性化、无偏聚合和更快收敛。

摘要:

使用低秩自适应(LoRA)对基础模型进行联邦微调为分布式学习提供了一种通信高效的解决方案。然而,现有的联邦LoRA方法存在两个基本限制:(1)结构聚合偏差,即独立平均低秩因子无法近似真实的组合更新;(2)客户端初始化滞后,即客户端在通信轮次中反复重新初始化LoRA参数,导致收敛变慢。我们提出HyperLoRA,一个统一的框架,通过超网络驱动的LoRA生成和乘积空间聚合的摊销联邦自适应来解决这两个问题。HyperLoRA不是进行迭代的逐客户端优化,而是使用一个学习到的生成器,将客户端分布特征映射到LoRA初始化,从而有效摊销每个客户端的自适应。在服务器端,我们引入一个学习到的聚合模块,直接在低秩乘积空间中合成更新,消除了因子级平均的不一致性。一个轻量级的残差校正模块进一步提高了在异质(非IID)客户端分布下的稳定性。通过用学习到的算子替代迭代优化和启发式平均,HyperLoRA共同实现了高效个性化、无偏聚合和更快的收敛。在联邦视觉和视觉-语言基准上的实验表明,与先前的联邦LoRA方法相比,HyperLoRA实现了更快的收敛速度、对分布偏移更强的鲁棒性以及更强的个性化性能。

英文摘要:

Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing convergence. We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation. Instead of iterative per-client optimization, HyperLoRA employs a learned generator that maps client distribution signatures to LoRA initializations, effectively amortizing per client adaptation. On the server side, we introduce a learned aggregation module that directly synthesizes updates in the low-rank product space, eliminating the inconsistencies of factor-wise averaging. A lightweight residual correction module further improves stability under heterogenous (non-IID) client distributions.By replacing iterative optimization and heuristic averaging with learned operators, HyperLoRA jointly enables efficient personalization, unbiased aggregation, and faster convergence. Experiments on federated vision and vision-language benchmarks show that HyperLoRA achieves improved convergence speed, greater robustness to distribution shift, and stronger personalization performance compared to prior federated LoRA methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑