PanoEnv:探索基于强化学习的全景环境中3D空间智能
PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning
AI总结:
PanoEnv通过强化学习框架提升VLMs在全景环境中3D空间推理能力,实现更高准确率和语义评分。
摘要:
360度全景图像越来越多地用于虚拟现实、自动驾驶和机器人领域,以实现对场景的整体理解。然而,当前的视觉-语言模型(VLMs)在等距投影(ERP)图像上的3D空间推理能力较差,原因是几何失真和有限的3D监督。我们引入了PanoEnv,一个大规模的VQA基准测试,该测试基于合成的3D环境构建,包含14800个问题,涵盖五个类别(例如相对位置、体积比较),并基于精确的3D注释,包括深度、分割和边界框。在对14种最先进的VLMs进行基准测试时,发现其3D理解能力有限,整体准确率为49.34%,开放式问题(OE)准确率为8.36%。为了增强3D推理能力,我们提出了一种基于组相对策略优化(GRPO)的强化学习后训练框架,结合了基于真实数据的奖励机制,整合了包括距离容忍和空间一致性在内的五种几何感知策略。一个两阶段课程进一步缓解了灾难性遗忘:阶段1训练于结构化任务(真假判断和多选),阶段2在混合开放式数据上进行微调以提高泛化能力。我们的7B模型实现了新的最先进的性能,整体准确率提高到52.93%(+3.59%),开放式问题准确率提高到14.83%,同时保持了结构化任务的性能。它还实现了顶级语义评估分数(Q-Score 6.24,P-Score 5.95),超过了32B模型。这些结果表明,PanoEnv-QA和我们基于课程的强化学习框架有效地在VLMs中灌输了3D空间智能,以实现全方位感知。
英文摘要:
360 panoramic images are increasingly used in virtual reality, autonomous driving, and robotics for holistic scene understanding. However, current Vision-Language Models (VLMs) struggle with 3D spatial reasoning on Equirectangular Projection (ERP) images due to geometric distortion and limited 3D supervision. We introduce PanoEnv, a large-scale VQA benchmark built from synthetic 3D environments, containing 14.8K questions across five categories (e.g., relative position, volume comparison) grounded in accurate 3D annotations including depth, segmentation, and bounding boxes. Benchmarking 14 state-of-the-art VLMs reveals limited 3D understanding, achieving only 49.34% overall accuracy and 8.36% on open-ended (OE) questions. To enhance 3D reasoning, we propose a reinforcement learning post-training framework based on Group Relative Policy Optimization (GRPO) with a ground-truth-guided reward that incorporates five geometry-aware strategies such as distance tolerance and spatial consistency. A two-stage curriculum further mitigates catastrophic forgetting: Stage 1 trains on structured tasks (true/false and multiple choice), and Stage 2 fine-tunes on mixed open-ended data to improve generalization. Our 7B model achieves new state-of-the-art performance, improving overall accuracy to 52.93% (+3.59%) and open-ended accuracy to 14.83% while maintaining structured-task performance. It also achieves top semantic evaluation scores (Q-Score 6.24, P-Score 5.95), surpassing 32B models. These results demonstrate that PanoEnv-QA and our curriculum-based RL framework effectively instill 3D spatial intelligence in VLMs for omnidirectional perception.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。