arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.03294cs.LGcs.AIcs.DC
收藏

DISTFLASHATTN:面向长上下文大语言模型训练的分布式内存高效注意力机制

DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training

UC Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);UCSD(加州大学圣地亚哥分校);CMU(卡内基梅隆大学);USC(南加州大学)

Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez, Hao Zhang

AI总结:

DISTFLASHATTN通过token级负载均衡、键值通信重叠和重物化感知梯度检查点,在长上下文LLM训练中显著扩展序列长度并实现多倍加速。

摘要:

FlashAttention(Dao, 2023)在单GPU上训练基于Transformer的大语言模型(LLMs)时,有效地将二次方峰值内存占用降低为线性。本文提出DISTFLASHATTN,一种针对长上下文LLMs训练优化的分布式内存高效注意力机制。我们提出三项关键技术:token级工作负载均衡、键值通信重叠,以及重物化感知的梯度检查点算法。我们在Llama-7B及其变体上评估DISTFLASHATTN,序列长度从32K到512K。与Ring Self-Attention相比,DISTFLASHATTN实现8倍更长序列和4.45至5.64倍加速;与使用FlashAttention的Megatron-LM相比,实现2至8倍更长序列和1.24至2.01倍加速。与近期的Ring Attention和DeepSpeed-Ulysses相比,分别实现1.67倍和1.26至1.88倍加速。代码见https://github.com/RulinShao/LightSeq。

英文摘要:

FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑