arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.02609cs.LGcs.AIcs.CLcs.DCstat.ML
收藏

Chronicals: 一种高性能的LLM微调框架,相比Unsloth提速3.51倍

Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth

Independent Researcher(独立研究者)

Arjun S. Nair

AI总结:

Chronicals通过四种协同优化,实现了比Unsloth快3.51倍的LLM微调速度,并提供了完整的数学基础和实现。

摘要:

大型语言模型的微调受到内存的限制:一个7B参数模型需要84GB-14GB用于权重,14GB用于梯度,56GB用于FP32优化器状态——甚至超过了A100-40GB的容量。我们提出了Chronicals,一种开源的训练框架,通过四种协同优化实现了比Unsloth快3.51倍的速度提升:(1)融合的Triton内核通过RMSNorm(7倍)、SwiGLU(5倍)和QK-RoPE(2.3倍)融合消除了75%的内存交通;(2)Cut Cross-Entropy通过在线softmax计算将logit内存从5GB减少到135MB;(3)LoRA+具有理论上推导出的16倍的适配器矩阵差分学习率;(4)Best-Fit Decreasing序列打包恢复了60-75%的计算浪费在填充上的部分。在Qwen2.5-0.5B上使用A100-40GB,Chronicals在完整微调中达到41,184 tokens/second,比Unsloth的11,736 tokens/second快3.51倍。对于LoRA在rank 32时,我们达到11,699 tokens/second,比Unsloth MAX的2,857 tokens/second快4.10倍。关键的是,我们发现Unsloth报告的46,000 tokens/second基准测试显示零梯度范数——模型并未在训练。我们提供了完整的数学基础:在线softmax正确性证明,FlashAttention IO复杂度界限O(N²d²M⁻¹),LoRA+学习率推导来自梯度幅度分析,以及装箱近似保证。所有实现、基准测试和证明都可在https://github.com/Ajwebdevs/Chronicals上获得,通过https://pypi.org/project/chronicals/进行pip安装。

英文摘要:

Large language model fine-tuning is bottlenecked by memory: a 7B parameter model requires 84GB--14GB for weights, 14GB for gradients, and 56GB for FP32 optimizer states--exceeding even A100-40GB capacity. We present Chronicals, an open-source training framework achieving 3.51x speedup over Unsloth through four synergistic optimizations: (1) fused Triton kernels eliminating 75% of memory traffic via RMSNorm (7x), SwiGLU (5x), and QK-RoPE (2.3x) fusion; (2) Cut Cross-Entropy reducing logit memory from 5GB to 135MB through online softmax computation; (3) LoRA+ with theoretically-derived 16x differential learning rates between adapter matrices; and (4) Best-Fit Decreasing sequence packing recovering 60-75% of compute wasted on padding. On Qwen2.5-0.5B with A100-40GB, Chronicals achieves 41,184 tokens/second for full fine-tuning versus Unsloth's 11,736 tokens/second (3.51x). For LoRA at rank 32, we reach 11,699 tokens/second versus Unsloth MAX's 2,857 tokens/second (4.10x). Critically, we discovered that Unsloth's reported 46,000 tokens/second benchmark exhibited zero gradient norms--the model was not training. We provide complete mathematical foundations: online softmax correctness proofs, FlashAttention IO complexity bounds O(N^2 d^2 M^{-1}), LoRA+ learning rate derivations from gradient magnitude analysis, and bin-packing approximation guarantees. All implementations, benchmarks, and proofs are available at https://github.com/Ajwebdevs/Chronicals with pip installation via https://pypi.org/project/chronicals/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑