RW-LoRA：基于随机游走的通信高效去中心化LoRA微调方法

中文摘要 AI 辅助

参数高效微调方法如LoRA已成为适配大基础模型的标准方式，将微调应用于分布式场景面临诸多挑战。现有多数分布式LoRA方法依赖集中式聚合，而基于 gossip 的去中心化 LoRA 则需要在多个模型副本间反复同步，两种方法均会产生显著通信开销，且因同时聚合多个模型更新引入误差。本文提出一种基于随机游走的 LoRA 微调方案，不维护多个模型副本，而是让单个模型令牌遍历网络，通过局部微调目标依次更新。该设计消除了全局同步需求，大幅降低通信与计算成本，避免聚合误差；在标准假设下，为非凸目标提供严格收敛保证。通过多个 NLP 任务和图拓扑的实证结果表明，该方法在实现可比任务性能的同时，相比基于 gossip 的 LoRA 大幅减少了通信与计算量。

英文摘要

Parameter-efficient fine-tuning methods such as LoRA have become a standard approach for adapting large foundation models. Adopting fine-tuning to distributed settings faces several challenges. Most existing distributed LoRA methods rely on centralized aggregation, and gossip-based decentralized LoRA requires repeated synchronization among multiple model copies. Both methods incur significant communication overhead and introduce errors due to simultaneous aggregation of multiple model updates. In this paper, we take a different perspective and propose a random-walk-based LoRA fine-tuning scheme. Instead of maintaining multiple model replicas, a single model token traverses the network and is updated sequentially using local fine-tuning objectives. This design eliminates the need for global synchronization, substantially reduces communication and computation costs, and avoids aggregation errors. We provide rigorous convergence guarantees for non-convex objectives under standard assumptions. Through empirical results on multiple NLP tasks and graph topologies, we show that the proposed method achieves competitive task performance with substantially less communication and computation than gossip-based LoRA.

发表机构

• Singapore University of Technology and Design（新加坡科技设计大学）
• LinkedIn（领英公司）
• Technical University of Munich（慕尼黑工业大学）
• Texas A&M University（德克萨斯农工大学）
• Rutgers University（罗格斯大学）

机构由 AI 辅助整理，请以论文原文为准。