arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.00654cs.CVcs.CLcs.LG
收藏

ICONS: 视觉-语言数据选择中的影响共识

ICONS: Influence Consensus for Vision-Language Data Selection

Princeton University(普林斯顿大学);University of Washington(华盛顿大学);Google DeepMind(谷歌DeepMind);Allen Institute for AI(人工智能研究院)

Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky

AI总结:

ICONS通过影响共识方法高效选择视觉-语言数据,保持高性能并支持多任务泛化。

摘要:

通过指令微调训练视觉-语言模型依赖于涵盖多种任务和领域的大量数据混合,但这些混合数据经常包含冗余信息,增加了计算成本而没有成比例的收益。现有方法通常依赖于任务无关的启发式方法来估计数据重要性,限制了其在不同任务上的有效性。我们介绍了ICONS,一种基于梯度的影响共识方法用于视觉-语言数据选择。我们的方法利用一阶训练动态来估计每个示例对验证性能的影响,然后通过多数投票在不同任务间聚合这些估计。这种跨任务共识识别出一致有价值的样本点,同时缓解了分数校准和异常值敏感性,使在不同多任务混合中实现稳健且可扩展的数据选择成为可能。在LLAVA-665K(分别:CAMBRIAN-7M,VISION-FLAN-186K)上训练的模型,使用我们选出的20%数据子集仍能保持98.6%(分别:98.8%,99.8%)的完整数据集性能。我们证明所选数据能够泛化到未见任务和模型架构,并发布了三个紧凑的子集LLAVA-ICONS-133K,CAMBRIAN-ICONS-1.4M,和VISION-FLAN-ICONS-37K用于高效的视觉-语言模型开发。

英文摘要:

Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heuristics to estimate data importance, limiting their effectiveness across tasks. We introduce ICONS, a gradient-based Influence CONsensus approach for vision-language data Selection. Our method leverages first-order training dynamics to estimate each example's influence on validation performance, then aggregates these estimates across tasks via majority voting. This cross-task consensus identifies consistently valuable data points while mitigating score calibration and outlier sensitivity, enabling robust and scalable data selection for diverse multitask mixtures. Models trained on our selected 20% data subset from LLAVA-665K (respectively: from CAMBRIAN-7M, from VISION-FLAN-186K) retain 98.6% (respectively: 98.8%, 99.8%) of full-dataset performance. We demonstrate that our selected data generalizes to unseen tasks and model architectures, and release three compact subsets LLAVA-ICONS-133K, CAMBRIAN-ICONS-1.4M, and VISION-FLAN-ICONS-37K for efficient vision-language model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑