arXiv 2606.285512026-08-11cs.CVcs.CLcs.LG

DataComp-VLM：改进的视觉语言模型开放数据集

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian Böther, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka St… 展开作者

Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian Böther, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

备注：: Preprint

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出DataComp-VLM基准，通过数据混合（而非过滤）策略提升视觉语言模型训练效果，在8B模型上达到63.6%准确率，比现有最优数据集提升5.4个百分点。

摘要：

构建高性能视觉语言模型（VLM）需要精心策划大规模训练数据集，然而社区缺乏系统性的基准来评估此类策划策略。我们引入了面向VLM的DataComp（DCVLM），这是一个用于受控数据中心实验的基准，旨在改进VLM训练。作为DCVLM的一部分，我们收集了涵盖四种数据类型（图像-文本对、多模态交错文档、纯文本和指令微调数据）的160个数据集，组成一个6T多模态token的语料库。DCVLM允许参与者测试跨1B-8B模型和6.25B-200B token预算的策划策略（过滤、混合、格式化、采样）。然后，模型在精心挑选的涵盖9个领域的多达52个下游基准上进行评估。我们在DCVLM上进行了大量实验，发现数据混合（而非过滤）是高质量训练数据集的关键：指令密集型混合比描述密集型混合扩展性更好，且增益在更大规模下扩大。由此产生的数据集DCVLM-Baseline，在使用200B训练token时，使8B VLM在33个任务核心套件上达到63.6%的准确率。与最先进的开放VLM训练数据集FineVision相比，这代表了+5.4个百分点的改进。DCVLM及其所有附带资源将在https://www.datacomp.ai/dcvlm/公开提供。

英文摘要：

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。