先筛选图像，后生成指令：面向视觉指令微调的指令前数据选择
Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning
AI总结：
针对视觉指令微调中从未标注图像生成指令成本高昂的问题，提出PreSel数据选择范式，通过评估任务重要性并聚类图像特征来筛选最具代表性的图像，仅对15%的图像生成指令，在大幅降低计算开销的同时达到与全量数据训练相当的性能。
摘要：
大型视觉语言模型（LVLMs）的视觉指令微调（VIT）需要在包含大量图像-指令对的数据集上进行训练，这可能会带来高昂的成本。近期在VIT数据选择方面的努力旨在选择一小部分高质量的图像-指令对子集，在保持与全规模训练相当的性能的同时减少VIT运行时间。然而，一个经常被忽视的主要挑战是，为VIT从未标注图像中生成指令非常昂贵。大多数现有的VIT数据集严重依赖人工标注或GPT API等付费服务，这限制了资源受限的用户为自定义应用创建VIT数据集。为了解决这个问题，我们引入了指令前数据选择（PreSel），这是一种更实用的数据选择范式，它直接选择最有益的未标注图像，并仅为选定的图像生成指令。PreSel首先估计VIT数据集中每个视觉任务的相对重要性，以得出任务级别的采样预算。然后它在每个任务内对图像特征进行聚类，在预算内选择最具代表性的图像。这种方法减少了VIT数据形成期间指令生成和LVLM微调的计算开销。通过仅为15%的图像生成指令，PreSel在LLaVA-1.5和Vision-Flan数据集上实现了与全数据VIT相当的性能。项目页面链接：https://bardisafa.github.io/PreSel
英文摘要：
Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets. The link to our project page: https://bardisafa.github.io/PreSel
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
