文本即图像：多模态大语言模型能否遵循像素中的印刷指令？
Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
AI总结：
本文提出 VIM 评测像素化印刷指令，在八个基准上揭示开源 MLLM 的显著性能下降，并训练可同时鲁棒遵循文本与视觉模态指令的 v-MLLM。
摘要：
近期的多模态大语言模型（MLLM）在视觉-语言任务上展现出良好的指令遵循能力。在本工作中，我们引入 VISUAL MODALITY INSTRUCTION（VIM，视觉模态指令），并考察多模态模型在预训练或微调期间未在此类数据上接受显式训练的情况下，能够多好地理解以像素形式提供的文本指令。我们将 VIM 适配到八个基准，包括 OKVQA、MM-Vet、MathVista 和 MMMU，并在文本模态指令（TEM）设置与 VIM 设置下探测多种 MLLM。值得注意的是，我们观察到开源 MLLM 在原始 TEM 与 VIM 设置之间存在显著性能差距，表明当文本指令仅以图像形式呈现时，开源 MLLM 面临更大挑战。为解决这一问题，我们训练了 v-MLLM，这是一个可泛化模型，能够在文本模态和视觉模态指令中执行鲁棒的指令遵循。
英文摘要：
Recent multimodal large language models (MLLMs) have shown promising instruction following capabilities on vision-language tasks. In this work, we introduce VISUAL MODALITY INSTRUCTION (VIM), and investigate how well multimodal models can understand textual instructions provided in pixels, despite not being explicitly trained on such data during pretraining or fine-tuning. We adapt VIM to eight benchmarks, including OKVQA, MM-Vet, MathVista, MMMU, and probe diverse MLLMs in both the text-modality instruction (TEM) setting and VIM setting. Notably, we observe a significant performance disparity between the original TEM and VIM settings for open-source MLLMs, indicating that open-source MLLMs face greater challenges when text instruction is presented solely in image form. To address this issue, we train v-MLLM, a generalizable model that is capable to conduct robust instruction following in both text-modality and visual-modality instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
