强化学习使多模态大语言模型比监督微调看得更清楚
RL makes MLLMs see better than SFT
AI总结：
本文研究了强化学习在多模态大语言模型中的视觉编码器影响，发现RL能产生更精确的视觉表示，提出PIVOT方法提升视觉编码性能。
摘要：
多模态语言模型（MLLM）研究中普遍认为其性能主要继承自LLM骨干网络，但视觉编码器的理解存在空白。本文通过对比监督微调（SFT）和强化学习（RL）的训练策略，发现RL在视觉问答任务中表现更优。通过图像分类、分割和梯度可视化等实验，发现RL能产生更精确的视觉表示，提出PIVOT方法，使视觉编码器在低计算成本下表现更佳。
英文摘要：
A dominant assumption in Multimodal Language Model (MLLM) research is that its performance is largely inherited from the LLM backbone, given its immense parameter scale and remarkable capabilities. This has created a void in the understanding of the vision encoder, which determines how MLLMs perceive images. The recent shift in MLLM training paradigms, from Supervised Finetuning (SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the significant lack of analysis on how such training reshapes the vision encoder as well as the MLLM. To address this, we first investigate the impact of training strategies on MLLMs, where RL shows a clear advantage over SFT in strongly vision-related VQA benchmarks. Motivated by this, we conduct a critical yet under-explored analysis of the vision encoder of MLLMs through diverse and in-depth experiments, ranging from ImageNet classification and segmentation to gradient visualization. Our results demonstrate that MLLM's post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual representations. Specifically, the key finding of our study is that RL produces stronger and precisely localized visual representations compared to SFT, boosting the ability of the vision encoder for MLLM. We then reframe our findings into a simple recipe for building strong vision encoders for MLLMs, Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs, a PIVOT-trained vision encoder outperforms even larger and more heavily-trained counterparts, despite requiring less than 1% of the computational cost of standard vision pretraining. This result opens an effective and efficient path for advancing the vision backbones of MLLMs. Project page available at https://june-page.github.io/pivot/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
