arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.21177cs.LGcs.CL
收藏

ChunkFT: 用于内存高效全微调的分块优化

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

Northeastern University, China(中国东北大学);Shanghai Jiao Tong University, China(上海交通大学);CIS, LMU Munich, Germany(慕尼黑大学CIS实验室);MCML, Germany(德国MCML实验室);Shandong University, China(山东大学)

Yongkang Liu, Zijing Wang, Mengjie Zhao, Ercong Nie, Mingyang Wang, Qian Li, Feiliang Ren, Shi Feng, Daling Wang, Hinrich Schütze

AI总结:

本文提出ChunkFT框架,通过动态激活的工作集重新定义全参数微调,实现了无需修改网络架构即可对任意子张量进行梯度计算,理论分析和实验表明其在内存使用、运行时间和优化质量上均有效,且在下游任务中表现优于现有内存高效基线。

摘要:

本文提出了ChunkFT,一种内存高效的微调框架,其通过动态激活的工作集重新定义全参数微调。ChunkFT能够在不修改网络架构的情况下,对任意子张量进行梯度计算,为优化任意子网络提供了算法基础,同时避免了标准密集梯度计算。在确定性设置下,我们提供了ChunkFT的理论收敛分析。实验中,我们使用单块RTX 4090-24GB GPU和两块H800-80GB GPU分别对Llama 3-8B和Llama 3-70B进行微调。一个7B模型在1K输入长度下的全参数微调仅需13.72GB的GPU内存。结果表明,ChunkFT在内存使用、运行时间和优化质量上均有效。此外,在语言理解、数学推理和MT-Bench等下游任务中,ChunkFT在性能上一致优于现有内存高效的基线。值得注意的是,ChunkFT在某些情况下甚至超过了全参数微调的性能。我们的代码库可在https://github.com/misonsky/chunk上找到。

英文摘要:

This work presents \textsc{ChunkFT}, a memory-efficient fine-tuning framework that reformulates full-parameter fine-tuning around a dynamically activated working set. \textsc{ChunkFT} enables gradient computation for arbitrary sub-tensors without modifying the network architecture, providing an algorithmic foundation for optimizing arbitrary sub-networks while avoiding standard dense gradient computation. We provide a theoretical convergence analysis of \textsc{ChunkFT} in the deterministic setting. Empirically, we apply \textsc{ChunkFT} to fine-tune Llama 3-8B and Llama 3-70B using a single RTX 4090-24GB GPU and 2$\times$ H800-80GB GPUs, respectively. Full-parameter fine-tuning of a 7B model with a 1K input length requires only 13.72GB of GPU memory. The results demonstrate the effectiveness of \textsc{ChunkFT} in memory usage, running time, and optimization quality. Moreover, downstream evaluations on language understanding, mathematical reasoning, and MT-Bench show that \textsc{ChunkFT} consistently outperforms existing memory-efficient baselines. Notably, \textsc{ChunkFT} achieves performance comparable to, and in some cases exceeding, full-parameter fine-tuning. Our repository is on https://github.com/misonsky/chunk.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑