arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.17847cs.CV
收藏

CoIDO:基于耦合重要性-多样性优化的视觉指令调优高效数据选择方法

CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization

The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Hangzhou Dianzi University(杭州电子科技大学);School of Computer Science (National Pilot Software Engineering School), BUPT(计算机学院(国家级试点软件工程学院),北京邮电大学)

Yichen Yan, Ming Zhong, Qi Zhu, Xiaoling Gu, Jinpeng Chen, Huan Li

AI总结:

针对视觉指令调优全量数据训练成本高、现有数据选择方法计算开销大且重要性与多样性分离优化的问题,提出双目标框架CoIDO,用轻量评分器和同方差不确定性平衡优化,仅用20%数据训练评分器筛选的20%子集,在LLaVA-1.5-7B十项任务上达全量微调98.2%的性能。

摘要:

多模态大语言模型(MLLMs)高度依赖指令调准来对齐视觉与语言能力,但在大规模数据集上训练的计算成本仍是主要瓶颈。现有数据选择方法试图通过筛选兼具重要性与多样性的子集来缓解这一问题,但存在两大关键缺陷:处理全量数据集带来的高计算开销,以及将重要性与多样性分开处理导致的数据选择效果欠佳。\n 我们提出了CoIDO,这是一种新型双目标框架,通过联合优化数据的重要性与多样性来克服上述挑战。现有方法通常需要对全量数据集进行成本高昂的评估,而CoIDO采用了一个轻量级的即插即用评分器。该评分器仅在少量随机采样数据上训练,用于学习候选集的分布,大幅降低了计算需求。通过利用基于同方差不确定性的公式,CoIDO在训练过程中有效平衡了重要性与多样性,实现了高效且可扩展的数据选择。\n 在实验中,我们仅使用20%的随机采样数据训练CoIDO评分器。训练完成后,将CoIDO应用于全量数据集,筛选出20%的子集用于指令调优。在广泛使用的LLaVA-1.5-7B模型的十项下游任务中,该筛选子集的平均性能达到了全量数据微调性能的98.2%,表现优异。

英文摘要:

Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity. We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling efficient and scalable data selection. In our experiments, we trained the CoIDO scorer using only 20 percent of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20 percent subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2 percent of the performance of full-data fine-tuning, on average.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑