视觉语言模型能多清楚地看见图像细节？
How Well Can Vision Language Models See Image Details?
AI总结：
本文提出像素值预测任务PVP，揭示VLM需适配视觉编码器才能准确预测像素值，并证明该预训练任务可显著提升细粒度视觉理解与游戏决策性能。
摘要：
基于大语言模型的视觉语言模型（LLM-based VLM）已在多种视觉语言理解任务中展现出令人瞩目的结果。然而，这些VLM在语义层面之外能多清楚地看见图像细节，仍不清楚。在本研究中，我们引入像素值预测任务（PVP），以探究“视觉语言模型能多清楚地看见图像细节？”，并帮助VLM感知更多细节。通常，这些模型由冻结的CLIP视觉编码器、大语言模型和连接模块组成。在PVP任务上微调VLM后，我们发现：1）仅微调连接模块和LLM时，现有VLM难以预测精确的像素值；2）当视觉编码器也被适配时，预测精度显著提高。此外，我们的研究揭示，将像素值预测作为VLM预训练任务之一并进行视觉编码器适配，可显著提升VLM在需要细致图像感知的下游图像-语言理解任务上的表现，例如指代表达图像分割（平均提升+10.19 cIoU）和视频游戏决策（在两款游戏上的平均得分分别提升+80.34和+70.54）。
英文摘要：
Large Language Model-based Vision-Language Models (LLM-based VLMs) have demonstrated impressive results in various vision-language understanding tasks. However, how well these VLMs can see image detail beyond the semantic level remains unclear. In our study, we introduce a pixel value prediction task (PVP) to explore "How Well Can Vision Language Models See Image Details?" and to assist VLMs in perceiving more details. Typically, these models comprise a frozen CLIP visual encoder, a large language model, and a connecting module. After fine-tuning VLMs on the PVP task, we find: 1) existing VLMs struggle to predict precise pixel values by only fine-tuning the connection module and LLM; and 2) prediction precision is significantly improved when the vision encoder is also adapted. Additionally, our research reveals that incorporating pixel value prediction as one of the VLM pre-training tasks and vision encoder adaptation markedly boosts VLM performance on downstream image-language understanding tasks requiring detailed image perception, such as referring image segmentation (with an average +10.19 cIoU improvement) and video game decision making (with average score improvements of +80.34 and +70.54 on two games, respectively).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
