arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.10620cs.DCcs.LG
收藏

DCP:通过动态上下文并行解决长上下文训练中的输入动态性问题

DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism

The University of Hong Kong(香港大学);Amazon Web Services(亚马逊网络服务)

Chenyu Jiang, Zhenkun Cai, Ye Tian, Zhen Jia, Yida Wang, Chuan Wu

AI总结:

提出动态上下文并行框架DCP,通过细粒度数据和计算分块及灵活映射,解决长上下文训练中序列长度和注意力模式动态变化导致的通信与计算不均衡问题,实现显著加速。

摘要:

上下文并行已成为支持长上下文训练的关键技术,这是现代大型模型生成式AI的发展趋势。然而,现有的上下文并行方法依赖静态并行配置,忽略了训练数据的动态特性,即不同样本间序列长度和token关系(即注意力模式)的变异性。因此,这些方法常导致不必要的通信开销和计算不均衡。本文提出DCP,一种动态上下文并行训练框架,引入数据和计算的细粒度分块划分。通过实现数据与计算块到设备的灵活映射,DCP能适应不同序列特性,有效减少通信并改善内存与计算平衡。微基准测试表明,DCP在因果掩码下将注意力加速1.19x~2.45x,在稀疏注意力模式下加速2.15x~3.77x。此外,我们观察到因果掩码下端到端训练加速达0.94x~1.16x,稀疏掩码下为1.00x~1.46x。

英文摘要:

Context parallelism has emerged as a key technique to support long-context training, a growing trend in generative AI for modern large models. However, existing context parallel methods rely on static parallelization configurations that overlook the dynamic nature of training data, specifically, the variability in sequence lengths and token relationships (i.e., attention patterns) across samples. As a result, these methods often suffer from unnecessary communication overhead and imbalanced computation. In this paper, we present DCP, a dynamic context parallel training framework that introduces fine-grained blockwise partitioning of both data and computation. By enabling flexible mapping of data and computation blocks to devices, DCP can adapt to varying sequence characteristics, effectively reducing communication and improving memory and computation balance. Micro-benchmarks demonstrate that DCP accelerates attention by 1.19x~2.45x under causal masks and 2.15x~3.77x under sparse attention patterns. Additionally, we observe up to 0.94x~1.16x end-to-end training speed-up for causal masks, and 1.00x~1.46x for sparse masks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑