通过自监督指导提升视觉指令微调
Boosting Visual Instruction Tuning with Self-Supervised Guidance
AI总结：
本文提出一种轻量方法，通过加入少量视觉基础自监督任务提升MLLMs的视觉推理能力，无需人工标注或架构修改，有效提升视觉中心评估性能。
摘要：
多模态大语言模型（MLLMs）在许多视觉-语言任务中表现良好，但往往在需要细粒度视觉推理的视觉中心问题上表现不佳。最近的证据表明，这一限制并非源于弱的视觉表示，而是由于在指令微调过程中对视觉信息的利用不足，许多任务可以通过语言先验单独部分解决。我们提出了一种简单且轻量的方法，通过加入少量视觉基础的自监督任务（以自然语言指令形式表达）来增强视觉指令微调。通过将经典的自监督预训练任务（如旋转预测、颜色匹配和跨视图对应）重新表述为图像-指令-响应三元组，我们引入了无法仅依赖视觉证据解决的监督。我们的方法不需要人工标注、架构修改或额外的训练阶段。在多个模型、训练方案和基准测试中，注入少量（3-10%）此类视觉基础指令始终提高了视觉中心评估的性能。我们的发现强调通过视觉基础自监督任务的指令微调作为提升MLLMs视觉推理能力的强大杠杆，通过简单调整训练数据分布实现。代码可在：https://github.com/sirkosophia/V-GIFT 获取。
英文摘要：
Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual reasoning in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
