arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21783cs.CV
收藏

学习在三维世界中思考与探索

Learning to Think and Explore in 3D World

University of California San Diego(加州大学圣地亚哥分校);Georgia Institute of Technology(佐治亚理工学院)

Yijiang Li, Bingyang Wang, Nuno Vasconcelos

AI总结:

该研究提出ACT3D模型,通过强化学习让视觉语言模型在三维空间中学习思考与探索,在六个三维基准测试中准确率达53.2%,优于基础Qwen模型、Gemini 3.6 Flash及多数开源三维方法,且泛化能力良好。

摘要:

我们研究视觉语言模型(VLMs)如何学习在三维空间中进行思考与主动探索。与向模型提供多视图图像、视频或三维表示不同,我们保留VLMs的原生视觉编码,即一次仅提供一张图像,就像人类一样,并将三维表示仅视为一个环境,模型可通过动作(例如向左移动)和观测(渲染图像)与该环境交互。每次轨迹从初始第一人称视角和一个视觉问题开始,模型随后可迭代地进行推理、导航和重新定向其视点,再给出最终答案。在初始的轨迹监督微调之后,我们应用强化学习(RL),其奖励主要由最终答案的正确性驱动。关键在于,RL并未对模型应访问的视点或应采取的探索动作提供直接监督,模型必须仅从其交互提升性能的程度中学习有效的探索策略。在六个三维基准测试中,我们的模型ACT3D达到了53.2%的平均准确率,较其基础Qwen模型提升了22.0%,并超过了专有模型Gemini 3.6 Flash。ACT3D在使用少50-70%输入帧的情况下,也优于现有的开源三维方法,且能泛化到未见过的领域,相比其监督初始化平均提升5.9%。进一步分析显示,模型会进行自适应的、基于问题的探索:在可从初始视图回答的问题上,较基础模型的性能提升为0.6%;在需要大量探索的问题上,提升则达16.3%。ACT3D会搜索被遮挡的目标、避免不必要的探索,并获取用于空间比较的互补视点。这些发现表明,基于结果的RL使VLMs不仅能对视觉证据进行推理,还能学习何时以及如何获取视觉证据。

英文摘要:

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑