arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01445cs.LGcs.CV
收藏

FastCLIP:一套在有限资源下加速 CLIP 训练的优化技术

FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

Texas A&M University(德克萨斯农工大学);Indiana University(印第安纳大学)

Xiyuan Wei, Fanjiang Ye, Ori Yonay, Xingyu Chen, Baixi Sun, Dingwen Tao, Tianbao Yang

AI总结:

本文提出面向分布式有限资源场景的 FastCLIP 框架,通过梯度归约及学习率、温度与参数更新优化,在最高32块GPU和多种数据规模下显著优于OpenCLIP。

摘要:

现有关于在大规模数据上训练最先进的对比语言-图像预训练(Contrastive Language-Image Pretraining,CLIP)模型的研究,由于需要大批量规模,往往涉及数百甚至数千块 GPU。然而,如此大量的资源对大多数人而言并不可得。尽管用于优化全局对比损失的先进组合优化技术已被证明能够消除对大批量规模的需求,但它们在大规模数据上的性能仍未得到充分探索和优化。为弥补这一差距,本文探索了在有限资源(例如至多数十块 GPU)下进行 CLIP 训练的若干方面。首先,我们提出 FastCLIP,这是一个基于先进组合优化技术构建、并为分布式环境设计和优化的通用 CLIP 训练框架。我们的框架配备了高效的梯度归约策略,以降低通信开销。其次,为进一步提升训练效率,我们从优化角度研究了该框架的三个组成部分:内部学习率的调度,以及温度参数和模型参数各自的更新规则。针对每个组成部分的不同策略所开展的实验,揭示了如何更高效地进行 CLIP 训练。最后,我们在最高达到 8 个节点上 32 块 GPU 的不同计算规模,以及从 270 万、910 万到 3.15 亿个图文对的三种数据规模上,对 FastCLIP 与最先进训练基线(OpenCLIP)的性能进行了基准测试,以证明 FastCLIP 在资源受限环境中的显著提升。我们在 https://github.com/Optimization-AI/fast_clip 发布 FastCLIP 的代码。

英文摘要:

Existing studies of training state-of-the-art Contrastive Language-Image Pretraining (CLIP) models on large-scale data involve hundreds of or even thousands of GPUs due to the requirement of a large batch size. However, such a large amount of resources is not accessible to most people. While advanced compositional optimization techniques for optimizing global contrastive losses have been demonstrated effective for removing the requirement of large batch size, their performance on large-scale data remains underexplored and not optimized. To bridge the gap, this paper explores several aspects of CLIP training with limited resources (e.g., up to tens of GPUs). First, we introduce FastCLIP, a general CLIP training framework built on advanced compositional optimization techniques while designed and optimized for the distributed setting. Our framework is equipped with an efficient gradient reduction strategy to reduce communication overhead. Second, to further boost training efficiency, we investigate three components of the framework from an optimization perspective: the schedule of the inner learning rate, the update rules of the temperature parameter and the model parameters, respectively. Experiments on different strategies for each component shed light on how to conduct CLIP training more efficiently. Finally, we benchmark the performance of FastCLIP and the state-of-the-art training baseline (OpenCLIP) on different compute scales up to 32 GPUs on 8 nodes, and three data scales ranging from 2.7 million, 9.1 million to 315 million image-text pairs to demonstrate the significant improvement of FastCLIP in the resource-limited setting. We release the code of FastCLIP at https://github.com/Optimization-AI/fast_clip .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑