arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.16979cs.CVcs.CL
收藏

DOSE: 通过现成模型进行多模态大语言模型的数据选择

DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models

Australian Artificial Intelligence Institute(澳大利亚人工智能研究所);Peking University(北京大学);University of Liverpool(利物浦大学)

Biao Wu, Yiwu Zhong, Meng Fang, Ling Chen

AI总结:

本文提出DOSE方法,利用现成预训练模型筛选多模态数据,提升数据质量和对齐性,减少计算成本,增强数据多样性,使模型在标准VQA和数学基准测试中表现优异。

摘要:

高质量和多样化的多模态数据对于提升视觉语言模型(VLMs)至关重要,但现有数据集常包含噪声、冗余和对齐不佳的样本。为解决这些问题,数据过滤常用于提高多模态学习的效率和性能,但过滤模型通常在相同数据上训练,导致额外计算成本。为减少此成本,我们研究DOSE,探索是否可以使用从未见过目标数据的现成预训练模型来选择更大更强的多模态模型的训练样本,而无需特定任务训练。即使不进行微调,这些模型也能有效评估文本质量和图像-文本对齐,以指导数据选择。基于此,我们构建了联合质量-对齐分布,并应用自适应加权采样来选择信息丰富的样本,同时保持长尾多样性。此方法增强了数据多样性,使在DOSE过滤数据上训练的模型在标准VQA和数学基准测试中可与全数据集训练的模型匹配或超越。大量实验证明了其有效性、效率和可扩展性。

英文摘要:

High-quality and diverse multimodal data are essential for improving vision-language models (VLMs), yet existing datasets often contain noisy, redundant, and poorly aligned samples. To address these problems, data filtering is commonly used to enhance the efficiency and performance of multimodal learning, but it introduces extra computational cost because filtering models are usually trained on the same data they are meant to screen. To reduce this cost, we study DOSE, which explores whether off-the-shelf pretrained models that have never seen the target data can be used to select training samples for larger and stronger multimodal models without any task-specific training. Even without fine-tuning, these models can effectively assess text quality and image-text alignment to guide data selection. Based on this, we build a joint quality-alignment distribution and apply adaptive weighted sampling to select informative samples while maintaining long-tail diversity. This approach enhances data diversity, enabling models trained on DOSE-filtered data to match or surpass those trained on the full dataset on standard VQA and math benchmarks. Extensive experiments demonstrate its effectiveness, efficiency, and scalability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑