arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10995cs.CVcs.LG
收藏

面向大型视觉-语言模型的基于概念-技能可迁移性的数据选择

Concept-skill Transferability-based Data Selection for Large Vision-Language Models

KAIST(韩国科学技术院);Nanyang Technological University(南洋理工大学);DeepAuto(深度自动科技(DeepAuto))

Jaewoo Lee, Boyang Li, Sung Ju Hwang

AI总结:

提出COINCIDE数据选择方法,利用小模型内部激活按概念-技能组合聚类并依据密度与可迁移性采样,使LVLM仅用少量数据即可达到接近全量微调的性能。

摘要:

指令微调,即在大量任务特定数据上进行监督微调,对于大型视觉-语言模型(LVLMs)在广泛的视觉-语言(VL)任务上实现良好泛化是必要的。然而,在大型VL数据集上训练可能会昂贵得令人却步。在本工作中,我们提出COINCIDE,这是一种有效且可扩展的数据选择技术,使用一个小模型作为参考模型来选择视觉指令微调数据,以高效微调目标LVLM,重点关注多样性和可迁移性。具体而言,我们利用小模型的内部激活对训练数据进行聚类,从而识别目标LVLM所需的VL概念-技能组合。随后,我们通过考虑这些多样聚类的密度和可迁移性(即良好迁移到其他概念-技能组合的能力)来从中采样数据。该方法确保了这些组合的多样性,而这对LVLM泛化至关重要。大量实验表明,在LLaVA-1.5和Vision-Flan两个不同数据集上,COINCIDE相较8个强基线取得了更优的性能和数据选择效率。仅使用LLaVA-1.5数据集的20%,COINCIDE即可达到与在整个数据集上微调的LVLM相当的性能,同时挂钟运行时间减少70%。在Vision-Flan数据集上,我们的方法仅使用16.7%的训练数据便取得了更优结果。

英文摘要:

Instruction tuning, or supervised finetuning on extensive task-specific data, is necessary for Large Vision-Language Models (LVLMs) to generalize well across a broad range of vision-language (VL) tasks. However, training on large VL datasets can become prohibitively expensive. In this work, we introduce COINCIDE, an effective and scalable data selection technique that uses a small model as a reference model to select visual instruction tuning data for efficient finetuning of a target LVLM, focusing on diversity and transferability. Specifically, we cluster the training data using internal activations from a small model, which identifies VL concept-skill compositions needed by a target LVLM. We then sample data from these diverse clusters by considering their density and transferability, or the ability to transfer well to other concept-skill compositions. This approach ensures the diversity of these compositions, which is vital for LVLM generalization. Extensive experiments demonstrate that COINCIDE achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan. Using only 20% of the LLaVA-1.5 dataset, COINCIDE achieves performance comparable to the LVLM finetuned on the whole dataset, with 70% reduction of the wall-clock running time. On the Vision-Flan dataset, our method achieves superior results with only 16.7% of the training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑