arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.16428cs.DCcs.AI
收藏

面向单GPU微调的高效异构协同设计

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

Hong Kong University of Science

Ruijia Yang, Zeyi Wen

AI总结:

提出SlideFormer系统,通过异步引擎、异构内存管理和优化Triton内核,在单GPU上高效微调超大规模语言模型,显著提升吞吐量并降低内存使用。

摘要:

微调大型语言模型(LLMs)已成为领域适应的关键,但其内存密集型特性超出了大多数GPU的能力。为应对这一挑战并推动LLM微调的普及,我们提出了SlideFormer,一种专为单GPU环境设计的新型系统。我们的创新包括:(1)轻量级异步引擎,将GPU视为滑动窗口,使GPU计算与CPU更新及多层I/O重叠执行;(2)高效的异构内存管理方案,显著降低峰值内存使用;(3)优化的Triton内核以解决关键瓶颈,并集成高级I/O。这种协同设计使得在单块RTX 4090上能够微调最新的123B+模型,支持高达8倍更大的批量大小和6倍更大的模型。评估中,SlideFormer相比基线实现了1.40倍至6.27倍的吞吐量提升,同时CPU/GPU内存使用大致减半,在NVIDIA和AMD GPU上均保持>95%的峰值性能。代码可在https://this https URL获取。

英文摘要:

Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.The code is available at https://github.com/RegiaYoung/SlideFormer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑