arXiv 2510.164162026-05-17cs.CVcs.AI

SSL4RL：重新审视自监督学习作为视觉语言推理的内在奖励

SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning

Peking University（北京大学）；MIT（麻省理工学院）；TUM（技术大学（TUM））；Meituan（美团）；Peking University School of CIT, MCML, MDSIEECS and CSAIL（北京大学计算机学院、MCML、MDSIEECS以及CSAIL）

Xiaojun Guo, Runyu Zhou, Yifei Wang, Qi Zhang, Chenheng Zhang, Stefanie Jegelka, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, Yisen Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SSL4RL框架，利用自监督学习任务作为强化学习的可验证奖励，提升视觉语言推理性能，并通过实验验证其在多模态模型对齐中的有效性。

摘要：

视觉语言模型（VLMs）通过整合大语言模型与视觉输入展现出显著能力，但往往无法充分利用视觉证据，要么依赖视觉任务中的语言先验，要么在推理中依赖文本捷径。尽管强化学习（RL）可以对齐模型与期望行为，但其在VLMs中的应用受限于缺乏可扩展且可靠的奖励机制。为克服这一挑战，我们提出了SSL4RL，一种新的框架，利用自监督学习（SSL）任务作为RL微调的可验证奖励源。我们的方法将SSL目标，如预测图像旋转或重建遮蔽块，转化为密集的自动奖励信号，消除了对人类偏好数据或不可靠的AI评估者的需求。实验表明，SSL4RL在视觉中心和视觉语言推理基准上显著提高了性能。此外，通过系统消融分析，我们识别出影响SSL4RL任务有效性的关键因素，如任务难度、模型规模和与目标领域语义的一致性，为未来工作提供了新的设计原则。我们还通过将其应用于图学习，展示了框架的通用性，其中取得了显著收益。SSL4RL建立了一种利用可验证的自监督目标对齐多模态模型的灵活且有效的方法论。

英文摘要：

Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequately, either depending on linguistic priors in vision-centric tasks or resorting to textual shortcuts during reasoning. Although reinforcement learning (RL) can align models with desired behaviors, its application to VLMs has been hindered by the lack of scalable and reliable reward mechanisms. To overcome this challenge, we propose SSL4RL, a novel framework that leverages self-supervised learning (SSL) tasks as a source of verifiable rewards for RL-based fine-tuning. Our approach reformulates SSL objectives-such as predicting image rotation or reconstructing masked patches-into dense, automatic reward signals, eliminating the need for human preference data or unreliable AI evaluators. Experiments show that SSL4RL substantially improves performance on both vision-centric and vision-language reasoning benchmarks. Furthermore, through systematic ablations, we identify key factors-such as task difficulty, model scale, and semantic alignment with the target domain-that influence the effectiveness of SSL4RL tasks, offering new design principles for future work. We also demonstrate the framework's generality by applying it to graph learning, where it yields significant gains. SSL4RL establishes a versatile and effective paradigm for aligning multimodal models using verifiable, self-supervised objectives.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。