arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.07461cs.CVcs.AI
收藏

BLIP3-KALE:知识增强的大规模密集描述

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

University of Washington(华盛顿大学);Salesforce Research(Salesforce研究院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校)
Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig S… 展开作者

Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig Schmidt, Yejin Choi, Caiming Xiong, Ran Xu

AI总结:

本文提出BLIP3-KALE数据集,通过两阶段方法利用大模型将合成密集描述与网络alt-text结合生成事实性描述,并训练专门VLM扩大数据集至2.18亿对,有效提升了多模态模型的视觉语言能力与知识性。

摘要:

我们介绍了BLIP3-KALE,这是一个包含2.18亿个图文对的数据集,它弥合了描述性合成描述与事实性网络规模alt-text之间的差距。KALE将网络规模alt-text融入合成密集图像描述中,以生成有事实依据的图像描述。我们的两阶段方法利用大型视觉-语言模型和语言模型创建知识增强描述,随后用其训练专门的VLM以扩大数据集规模。我们在KALE上训练视觉-语言模型,并展示了在视觉-语言任务上的提升。实验表明KALE在训练更强大、更具知识性的多模态模型方面的效用。我们在https://huggingface.co/datasets/Salesforce/blip3-kale发布KALE数据集。

英文摘要:

We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑