基于闭环VLM代理的文本引导的6D物体姿态重排
Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents
AI总结：
本文提出通过闭环VLM代理实现文本引导的6D物体姿态重排，引入多视角推理、坐标系可视化和单轴旋转预测等技术，提升VLM在3D场景中推理目标物体姿态的能力，且在不同VLM上均有效。
摘要：
视觉-语言模型（VLMs）表现出强大的视觉推理能力，但仍然在3D理解方面存在困难。特别是，VLMs常常无法推断3D场景中目标物体的一致文本目标6D姿态。然而，我们发现通过一些推理时间技术和迭代推理，VLMs可以实现显著的性能提升。具体而言，给定一个由RGB-D图像（或3D网格组成场景）表示的3D场景和一个指定所需状态变化的文本指令，我们重复以下循环：观察当前场景；评估其是否忠实于指令；提出目标物体的姿态更新；应用更新；并渲染更新后的场景。通过这种闭环交互，VLMs有效地充当了代理。我们进一步引入三种推理时间技术，这些技术对于此闭环过程至关重要：（i）多视角推理与辅助视角选择，（ii）以物体为中心的坐标系可视化，以及（iii）单轴旋转预测。在没有任何额外微调或新模块的情况下，我们的方法在预测文本引导的目标6D姿态方面超过了先前的方法。它在封闭源和开源VLMs上均表现一致。此外，当将我们的6D姿态预测与简单的机器人运动规划结合时，它能够实现比现有方法更成功的机器人操作。最后，我们进行了一项消融研究，以证明每个提出技术的必要性。
英文摘要：
Vision-Language Models (VLMs) exhibit strong visual reasoning capabilities, yet they still struggle with 3D understanding. In particular, VLMs often fail to infer a text-consistent goal 6D pose of a target object in a 3D scene. However, we find that with some inference-time techniques and iterative reasoning, VLMs can achieve dramatic performance gains. Concretely, given a 3D scene represented by an RGB-D image (or a compositional scene of 3D meshes) and a text instruction specifying a desired state change, we repeat the following loop: observe the current scene; evaluate whether it is faithful to the instruction; propose a pose update for the target object; apply the update; and render the updated scene. Through this closed-loop interaction, the VLM effectively acts as an agent. We further introduce three inference-time techniques that are essential to this closed-loop process: (i) multi-view reasoning with supporting view selection, (ii) object-centered coordinate system visualization, and (iii) single-axis rotation prediction. Without any additional fine-tuning or new modules, our approach surpasses prior methods at predicting the text-guided goal 6D pose of the target object. It works consistently across both closed-source and open-source VLMs. Moreover, when combining our 6D pose prediction with simple robot motion planning, it enables more successful robot manipulation than recent Vision-Language-Action models (VLAs). Finally, we conduct an ablation study to demonstrate the necessity of each proposed technique.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
