arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.04598cs.LGcs.AIcs.CV
收藏

面向开放基础语言-视觉模型与数据集鲁棒比较的缩放定律

Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets

LAION;Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)(联合计算中心(JSC),焦耳希中心(FZJ));Eindhoven University of Technology(埃因霍温理工大学)

Marianna Nezhurina, Tomer Porian, Giovanni Pucceti, Tommie Kerssies, Romain Beaumont, Mehdi Cherti, Jenia Jitsev

AI总结:

本文首次为 CLIP 和 MaMMUT 两种语言-视觉模型推导完整缩放定律,用于跨规模比较模型与数据集,证明 MaMMUT 在规模扩展和样本效率上优于 CLIP,并发布 openMaMMUT-L/14 模型及实验资源。

摘要:

在可迁移学习研究中,缩放定律被用于多种重要基础模型,以预测其在更大规模下的性质与性能。本文展示了缩放定律推导也可用于模型与数据集比较,从而判断预训练时应优先采用哪种方案。我们首次基于在广泛的模型规模和样本规模上的密集测量,为两种重要的语言-视觉学习流程 CLIP 和 MaMMUT 推导出完整缩放定律,其中 CLIP 仅使用对比损失,而 MaMMUT 同时使用对比损失和图像描述文本生成损失。在确保对留出数据点具有足够预测精度的前提下,我们利用推导出的缩放定律比较两种模型,得到证据表明 MaMMUT 随规模增长具有更强的性能提升,并且比标准 CLIP 具有更好的样本效率。为增强比较的有效性,我们展示了多个下游任务上的缩放定律,包括分类、检索和分割,并在不同开放数据集 DataComp、DFN 和 Re-LAION 上进行验证,观察到一致的趋势。我们还表明,在使用恒定学习率调度推导缩放定律时同样可以进行比较,从而降低计算成本。因此,精确推导缩放定律为跨规模范围进行模型和数据集比较提供了手段,避免仅基于单一参考规模的测量得出误导性结论,为系统比较和改进开放基础模型及数据集铺平了道路。我们发布了所有预训练模型及其中间检查点,包括 openMaMMUT-L/14,该模型在 DataComp-1.4B 的 128 亿样本上训练,达到 80.3% 的零样本 ImageNet-1k 准确率。用于复现论文实验的代码和原始实验数据可在 https://github.com/LAION-AI/scaling-laws-for-comparison 找到。

英文摘要:

In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. We show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. For the first time, full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. Ensuring sufficient prediction accuracy for held out points, we use derived scaling laws to compare both models, obtaining evidence for MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, classification, retrieval, and segmentation, and for different open datasets, DataComp, DFN and Re-LAION, observing consistently the same trends. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison across scale spans, avoiding misleading conclusions based on measurements from single reference scales only, paving the road for systematic comparison and improvement of open foundation models and datasets for their creation. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUT-L/14, which achieves $80.3\%$ zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/scaling-laws-for-comparison.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑