面向未来的提示：基于交互式数字孪生的开放世界模型预测控制
Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
AI总结：
针对视觉语言模型物理理解不足、难以预测低层机器人控制的问题，提出融合VLM语义推理与交互式数字孪生的开放世界模型预测控制框架，通过模拟生成轨迹、未来提示选优及多视点合成提升性能，在复杂操作任务上优于基线方法。
摘要：
开放世界机器人操作领域的最新进展很大程度上由视觉语言模型（VLM）推动。尽管这些模型在高层规划中展现出强大的泛化能力，但由于对物理世界的理解有限，它们难以预测低层机器人控制指令。为解决这一问题，我们提出了一种面向开放世界操作的模型预测控制框架，该框架将VLM的语义推理能力与真实环境的物理接地交互式数字孪生相结合。通过构建并模拟数字孪生，我们的方法生成可行的运动轨迹，模拟相应结果，并向VLM提供未来观测结果作为提示，以根据任务的语言指令评估并选择最合适的结果。为进一步增强预训练VLM理解复杂场景以用于机器人控制的能力，我们利用数字孪生的灵活渲染能力，从各种新颖的无遮挡视点合成场景。我们在一系列多样化的复杂操作任务上验证了我们的方法，证明其相较于使用VLM的语言条件机器人控制基线方法具有更优性能。
英文摘要：
Recent advancements in open-world robot manipulation have been largely driven by vision-language models (VLMs). While these models exhibit strong generalization ability in high-level planning, they struggle to predict low-level robot controls due to limited physical-world understanding. To address this issue, we propose a model predictive control framework for open-world manipulation that combines the semantic reasoning capabilities of VLMs with physically-grounded, interactive digital twins of the real-world environments. By constructing and simulating the digital twins, our approach generates feasible motion trajectories, simulates corresponding outcomes, and prompts the VLM with future observations to evaluate and select the most suitable outcome based on language instructions of the task. To further enhance the capability of pre-trained VLMs in understanding complex scenes for robotic control, we leverage the flexible rendering capabilities of the digital twin to synthesize the scene at various novel, unoccluded viewpoints. We validate our approach on a diverse set of complex manipulation tasks, demonstrating superior performance compared to baseline methods for language-conditioned robotic control using VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
