面向大规模视觉-语言模型预训练的低幻觉合成字幕
Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
AI总结：
针对视觉-语言预训练高质量图文数据稀缺的问题，提出基于连续DPO的低幻觉合成字幕生成流水线，可替代真实数据并显著提升多模态模型及文生图模型的性能。
摘要：
近年来，受大语言模型文本能力持续提升的推动，视觉-语言模型预训练领域取得了快速进展。然而，现有的多模态大语言模型训练范式严重依赖高质量的图文对。随着模型规模和数据规模呈指数级增长，这类经过精心整理的数据的可用量日益稀缺且趋于饱和，严重制约了该领域的进一步发展。本研究探索了面向视觉-语言模型预训练的可扩展字幕生成技术，并证明大规模低幻觉合成字幕具有双重作用：1）可作为预训练范式中真实世界数据的可行替代方案；2）经实证验证，将其融入视觉-语言模型后可实现更优的性能提升。本文的主要贡献如下：1）提出了一种生成高质量、低幻觉、知识丰富的合成字幕的新型流水线。我们的连续DPO（直接偏好优化）方法在减少幻觉方面效果显著，具体而言，对于7B规模的模型，在留出测试集上的无幻觉字幕率从48.3%提升至77.9%。2）全面的实证验证表明，我们的合成字幕相比同类字幕具备更优的预训练优势：在15项视觉-语言任务上，使用我们的数据训练的模型，与使用相同图像的替代文本（alt-text）训练的模型相比，性能提升至少6.2%；在20个常见认知领域中，前者的表现比后者高出至少7.5%。同时，该合成字幕在文本生成图像领域也提供了显著支持：使用我们的数据集后，在真实世界验证基准上的FID分数降低了17.1，在MSCOCO验证基准上的FID分数降低了13.3。
英文摘要：
In recent years, the field of vision-language model pre-training has experienced rapid advancements, driven primarily by the continuous enhancement of textual capabilities in large language models. However, existing training paradigms for multimodal large language models heavily rely on high-quality image-text pairs. As models and data scales grow exponentially, the availability of such meticulously curated data has become increasingly scarce and saturated, thereby severely limiting further advancements in this domain. This study investigates scalable caption generation techniques for vision-language model pre-training and demonstrates that large-scale low-hallucination synthetic captions can serve dual purposes: 1) acting as a viable alternative to real-world data for pre-training paradigms and 2) achieving superior performance enhancement when integrated into vision-language models through empirical validation. This paper presents following key contributions: 1) a novel pipeline for generating high-quality, low-hallucination, and knowledge-rich synthetic captions. Our continuous DPO methodology yields remarkable results in reducing hallucinations. Specifically, the non-hallucination caption rate on a held-out test set increases from 48.3% to 77.9% for a 7B-size model. 2) Comprehensive empirical validation reveals that our synthetic captions confer superior pre-training advantages over their counterparts. Across 15 vision language tasks, the model trained with our data achieves a significant performance gain of at least 6.2% compared to identical images with alt-text. In 20 common cognitive domains, the model trained with our data outperforms the alt-text data by at least 7.5%. Meanwhile, it also offers considerable support in the text-to-image domain. With our dataset, the FID score is reduced by 17.1 on a real-world validation benchmark and 13.3 on the MSCOCO validation benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
