arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.16166cs.CVcs.CL
收藏

超越过滤:面向多模态大语言模型预训练的自适应图像-文本质量增强

Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining

University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Baichuan Inc.(百川智能公司);New Laboratory of Pattern Recognition (NLPR)(新模式识别实验室);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)

Han Huang, Yuqi Huo, Zijia Zhao, Haoyu Lu, Shu Wu, Bingning Wang, Qiang Liu, Weipeng Chen, Liang Wang

AI总结:

本文提出自适应图像-文本质量增强器AITQE,通过文本重写和负样本学习动态提升低质量图文对质量,在保留数据量的同时超越现有过滤方法,实现高效可扩展的MLLM预训练。

摘要:

多模态大语言模型(MLLMs)通过整合视觉与文本模态取得了显著进展。训练MLLMs的一个关键因素是多模态预训练数据集中图像-文本对的质量。然而,事实上的基于过滤的数据质量增强范式常常由于图像与文本之间语义对齐不足而丢弃大量高质量图像数据,导致数据利用效率和可扩展性低下。在本文中,我们提出了自适应图像-文本质量增强器(AITQE),这是一个动态评估和增强图像-文本对质量的模型。AITQE对低质量对采用文本重写机制,并引入负样本学习策略,通过在训练过程中整合刻意选择的低质量样本来提升评估能力。与先前显著改变文本分布的方法不同,我们的方法对文本进行最小调整,以在提升质量的同时保留数据量。实验结果表明,AITQE在各种基准上超越了现有方法,有效利用了原始数据,并随着数据量的增加高效扩展。我们希望我们的工作能启发未来的研究。代码和模型可在以下地址获取:https://github.com/hanhuang22/AITQE。

英文摘要:

Multimodal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets. However, $\textit {de facto}$ filter-based data quality enhancement paradigms often discard a substantial portion of high-quality image data due to inadequate semantic alignment between images and texts, leading to inefficiencies in data utilization and scalability. In this paper, we propose the Adaptive Image-Text Quality Enhancer (AITQE), a model that dynamically assesses and enhances the quality of image-text pairs. AITQE employs a text rewriting mechanism for low-quality pairs and incorporates a negative sample learning strategy to improve evaluative capabilities by integrating deliberately selected low-quality samples during training. Unlike prior approaches that significantly alter text distributions, our method minimally adjusts text to preserve data volume while enhancing quality. Experimental results demonstrate that AITQE surpasses existing methods on various benchmark, effectively leveraging raw data and scaling efficiently with increasing data volumes. We hope our work will inspire future works. The code and model are available at: https://github.com/hanhuang22/AITQE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑