Molmo和PixMo：面向最先进视觉语言模型的开放权重与开放数据
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
AI总结：
针对社区缺乏从零构建高性能VLMs基础认知的问题，本文提出开放VLM家族Molmo及未使用外部VLMs的PixMo数据集集合，其72B模型在开放类别中性能领先，还优于多款更大的专有模型，仅次于GPT-4o。
摘要：
当前最先进的视觉语言模型（VLMs）仍为专有模型。性能最强的开放权重模型严重依赖专有VLMs生成的合成数据来实现良好性能，实际上是将这些闭源VLMs蒸馏为开源模型。因此，社区一直缺乏关于如何从零开始构建高性能VLMs的基础认知。我们提出Molmo，这是一类在开放性类别中达到最先进水平的VLM家族。我们的核心贡献是名为PixMo的新数据集集合，包括用于预训练的高细节图像描述数据集、用于微调的自由形式图像问答数据集，以及创新的2D指向数据集，所有数据集均未使用外部VLMs收集。我们方法的成功依赖于精心的建模选择、优化的训练流程，以及最关键的新收集数据集的质量。我们的最佳72B模型不仅在开放权重和开放数据模型类别中优于其他模型，还在学术基准和大规模人类评估中优于更大的专有模型（包括Claude 3.5 Sonnet、Gemini 1.5 Pro和Flash），仅次于GPT-4o。我们的模型权重、新数据集和源代码可在https://molmo.allenai.org/blog获取。
英文摘要：
Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
