arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.13847cs.LGcs.AI
收藏

SparseBalance: 基于动态稀疏注意力的负载平衡长上下文训练

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Meituan(美团);University of Chinese Academy of Sciences(中国科学院大学)

Hongtao Xu, Jianchao Tan, Yuxuan Hu, Pengju Lu, Hongyu Wang, Pingwei Sun, Yerui Sun, Yuchen Xie, Xunliang Cai, Mingzhen Li, Weile Jia

AI总结:

本文提出SparseBalance算法系统协同设计框架,通过动态稀疏调整和稀疏感知批处理策略,解决长上下文训练中的负载不平衡问题,提升模型精度与系统效率。

摘要:

尽管稀疏注意力缓解了长上下文LLM训练的计算瓶颈,但其分布式训练过程在序列长度和稀疏敏感性方面表现出极端异质性,导致严重的不平衡问题和次优模型精度。现有算法和训练框架通常只关注单一问题,未能系统性地优化这两个问题。因此,我们提出SparseBalance,一种新颖的算法-系统协同设计框架,利用稀疏性和序列异质性,联合优化模型精度和系统效率。首先,我们提出工作负载感知的动态稀疏调整,通过双向稀疏调整消除滞后者并利用内在气泡提升精度。其次,我们提出稀疏感知的批处理策略以实现粗粒度平衡,该策略补充动态稀疏调整。实验结果表明,SparseBalance在LongBench基准上提升了长上下文能力0.46%,并实现了高达1.33倍的端到端加速。

英文摘要:

While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both \textit{1)} sequence length and \textit{2)} sparsity sensitivity, leading to a severe imbalance problem and sub-optimal model accuracy. Existing algorithms and training frameworks typically focus on single issue, failing to systematically co-optimize these two problems. Therefore, we propose SparseBalance, a novel algorithm-system co-design framework, which exploits the sparsity and sequence heterogeneity to optimize model accuracy and system efficiency jointly. First, we propose workload-aware dynamic sparsity tuning, which employs a bidirectional sparsity adjustment to eliminate stragglers and exploit inherent bubbles for free accuracy. Second, we propose a sparsity-aware batching strategy to achieve coarse-grained balance, which complements dynamic sparsity tuning. Experimental results demonstrate that SparseBalance achieves up to a 1.33$\times$ end-to-end speedup while still improving the long-context capability by 0.46\% on the LongBench benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑