arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.04155cs.CVcs.CLcs.LG
收藏

MM-GEN:通过有针对性的多模态数据策划提升任务性能

MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

Microsoft Research(微软研究院);UCLA(加利福尼亚大学洛杉矶分校);UIUC(伊利诺伊大学厄巴纳-香槟分校)

Siddharth Joshi, Besmira Nushi, Vidhisha Balachandran, Varun Chandrasekaran, Vibhav Vineet, Neel Joshi, Baharan Mirzasoleiman

AI总结:

本文提出MM-Gen多模态合成数据策划方法,通过分组、定向生成与过滤为VLM构建任务专用训练文本,显著提升Llava-1.5等模型的空间推理和图解理解性能。

摘要:

视觉-语言模型(VLM)非常有效,但在专门任务上往往表现不佳;例如,由于缺乏任务特定的训练数据,Llava-1.5在图表和图解理解方面存在困难。现有训练数据来源于通用数据集,无法捕捉这些任务所需的细微细节。我们提出MM-Gen,这是一种可扩展的方法,通过利用更强的模型为候选图像生成任务特定的高质量合成文本。MM-Gen采用三阶段有针对性的流程:将数据划分为子组,根据任务描述生成有针对性的文本,并过滤冗余和离群数据。使用MM-Gen生成的数据微调VLM可带来显著的性能提升,包括Llava-1.5(7B)在空间推理上提升29%,在图解理解上提升15%。与人工策划的字幕数据相比,MM-Gen对原始模型的改进最高可达1.6倍,证明了其在增强任务特定VLM性能以及弥合通用数据集与专门需求之间差距方面的有效性。代码可在https://github.com/sjoshi804/MM-Gen获取。

英文摘要:

Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1.5 struggles with chart and diagram understanding due to scarce task-specific training data. Existing training data, sourced from general-purpose datasets, fails to capture the nuanced details needed for these tasks. We introduce MM-Gen, a scalable method that generates task-specific, high-quality synthetic text for candidate images by leveraging stronger models. MM-Gen employs a three-stage targeted process: partitioning data into subgroups, generating targeted text based on task descriptions, and filtering out redundant and outlier data. Fine-tuning VLMs with data generated by MM-Gen leads to significant performance gains, including 29% on spatial reasoning and 15% on diagram understanding for Llava-1.5 (7B). Compared to human-curated caption data, MM-Gen achieves up to 1.6x better improvements for the original models, proving its effectiveness in enhancing task-specific VLM performance and bridging the gap between general-purpose datasets and specialized requirements. Code available at https://github.com/sjoshi804/MM-Gen.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑