arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02677cs.CVcs.CL
收藏

经过微调的多模态语言模型是高质量的图文数据过滤器

Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Bytedance(字节跳动)

Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, Heng Wang

AI总结:

本文提出利用微调后的多模态语言模型构建图文数据过滤框架,通过四项互补指标和高质量指令数据流程,替代CLIPScore并提升基础模型在多任务上的表现。

摘要:

我们提出了一个新颖的框架,通过利用经过微调的多模态语言模型(Multimodal Language Models,MLM)来过滤图文数据。通过整合MLM的最新进展,我们的方法优于主流过滤方法(例如 CLIPScore)。我们设计了四个不同但互补的指标,以全面衡量图文数据的质量。我们还建立了一条新的流程,用于构建高质量指令数据,从而将MLM微调为数据过滤器。与CLIPScore相比,我们的MLM过滤器能够产生更精确、更全面的分数,直接提升被过滤数据的质量,并提高预训练模型的性能。在流行的基础模型(即CLIP和BLIP2)以及多种下游任务上,我们相较CLIPScore取得了显著改进。我们的MLM过滤器能够泛化到不同模型和任务,并可作为CLIPScore的即插即用替代方案。本文还提供了额外的消融研究,以验证我们针对MLM过滤器的设计选择。

英文摘要:

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑