我们真的需要视觉指令吗？迈向大型视觉-语言模型的无视觉指令微调
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
AI总结：
针对视觉指令微调依赖图文数据、难以继承 LLM 能力的问题，提出 ViFT 框架，用纯文本指令和图像描述分别训练两类能力并在推理时融合，以更少数据达到先进性能。
摘要：
视觉指令微调已成为激发大型视觉-语言模型（LVLMs）多模态任务解决能力的主流技术。尽管取得了成功，但由于视觉指令需要图像作为输入，这会在继承骨干 LLM 的任务解决能力方面留下差距，并使大规模数据集的收集成本高昂。为解决该问题，我们提出 ViFT，这是一种面向 LVLMs 的无视觉指令微调框架。在 ViFT 中，训练期间我们只需要纯文本指令和图像描述数据，以分别学习任务解决能力和视觉感知能力。推理期间，我们提取并组合文本与图像输入的表示，用于融合这两种能力以完成多模态任务。实验结果表明，ViFT 能在多个视觉推理和视觉指令遵循基准上以少得多的训练数据达到最先进性能。我们的代码和数据将公开发布。
英文摘要：
Visual instruction tuning has become the predominant technology in eliciting the multimodal task-solving capabilities of large vision-language models (LVLMs). Despite the success, as visual instructions require images as the input, it would leave the gap in inheriting the task-solving capabilities from the backbone LLMs, and make it costly to collect a large-scale dataset. To address it, we propose ViFT, a visual instruction-free fine-tuning framework for LVLMs. In ViFT, we only require the text-only instructions and image caption data during training, to separately learn the task-solving and visual perception abilities. During inference, we extract and combine the representations of the text and image inputs, for fusing the two abilities to fulfill multimodal tasks. Experimental results demonstrate that ViFT can achieve state-of-the-art performance on several visual reasoning and visual instruction following benchmarks, with rather less training data. Our code and data will be publicly released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
