arXiv 2602.005042026-01-31cs.CV

RGBX-R1: 多视觉模态链式推理引导的多模态接地强化学习

RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding

School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；Low-Altitude Intelligence Lab, Xiong’an National Innovation Center（雄安国家创新中心低空智能实验室）；Xiong’an Guochuang Lantian Technology Co., Ltd.（雄安国创莲田科技有限公司）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学学院）

Jiahe Wu, Bing Cao, Qilong Wang, Qinghua Hu, Dongdong Li, Pengfei Zhu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RGBX-R1通过视觉模态链式推理引导的强化学习提升多模态接地能力，首次构建RGBX-接地基准并在多个任务中取得显著优势。

摘要：

多模态大语言模型（MLLM）主要在RGB模态上进行预训练，从而限制了其在其他模态，如红外、深度和事件数据上的性能，这些模态对于复杂场景至关重要。为了解决这一问题，我们提出了RGBX-R1框架，旨在增强MLLM在各种X视觉模态上的感知和推理能力。具体来说，我们采用理解-关联-验证（UAV）提示策略来构建视觉模态链式推理（VM-CoT），旨在将MLLM的RGB理解能力扩展到X模态。为了逐步增强推理能力，我们引入了双阶段训练范式：冷启动监督微调（CS-SFT）和时空强化微调（ST-RFT）。CS-SFT通过VM-CoT的指导监督推理过程，使MLLM具备基本的模态认知能力。在此基础上，ST-RFT采用模态理解时空（MuST）奖励来强化模态推理。值得注意的是，我们构建了首个RGBX-接地基准，大量实验验证了我们在多模态理解和空间感知上的优势，在三个RGBX接地任务上，比基线高出22.71%。

英文摘要：

Multimodal Large Language Models (MLLM) are primarily pre-trained on the RGB modality, thereby limiting their performance on other modalities, such as infrared, depth, and event data, which are crucial for complex scenarios. To address this, we propose RGBX-R1, a framework to enhance MLLM's perception and reasoning capacities across various X visual modalities. Specifically, we employ an Understand-Associate-Validate (UAV) prompting strategy to construct the Visual Modality Chain-of-Thought (VM-CoT), which aims to expand the MLLMs' RGB understanding capability into X modalities. To progressively enhance reasoning capabilities, we introduce a two-stage training paradigm: Cold-Start Supervised Fine-Tuning (CS-SFT) and Spatio-Temporal Reinforcement Fine-Tuning (ST-RFT). CS-SFT supervises the reasoning process with the guidance of VM-CoT, equipping the MLLM with fundamental modality cognition. Building upon GRPO, ST-RFT employs a Modality-understanding Spatio-Temporal (MuST) reward to reinforce modality reasoning. Notably, we construct the first RGBX-Grounding benchmark, and extensive experiments verify our superiority in multimodal understanding and spatial perception, outperforming baselines by 22.71% on three RGBX grounding tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。