arXiv 2506.109662025-06-12cs.RO

GENMANIP：面向可泛化指令跟随操作的大语言模型驱动仿真

GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation

Shanghai AI Laboratory（上海人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；Zhejiang University（浙江大学）；Nanjing University（南京大学）

Ning Gao, Yilun Chen, Shuai Yang, Xinyi Chen, Yang Tian, Hao Li, Haifeng Huang, Hanqing Wang, Tai Wang, Jiangmiao Pang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对机器人操作泛化难、现有仿真平台支持不足的问题，研究推出GenManip仿真平台与GenManip-Bench基准，实验表明带基础模型的模块化策略泛化效果优于数据增强的端到端策略。

摘要：

真实场景下的机器人操作仍具挑战性，尤其是在鲁棒泛化方面。现有仿真平台对探索策略如何适配多样指令与场景的支持不足，因此跟不上人们对LLM（大语言模型）这类指令跟随基础模型日益增长的研究兴趣——这类模型的适应性至关重要，但在公平对比中仍未得到充分探究。为填补这一空白，我们推出GenManip，一款专为策略泛化研究打造的高真实度桌面仿真平台。它具备由LLM驱动的面向任务的场景图自动生成管线，可利用1万个带标注的3D物体资产合成大规模、多样化的任务。为系统性评估泛化能力，我们提出GenManip-Bench基准，包含200个经人在回路校正优化的场景。我们评估了两类策略：（1）整合基础模型用于感知、推理与规划的模块化操作系统；（2）通过可扩展数据收集训练的端到端策略。结果显示，尽管数据规模扩大会让端到端方法受益，但搭载基础模型增强的模块化系统在多样场景中的泛化效果更优。我们期待该平台能为推进真实条件下的策略泛化研究提供关键洞见。项目主页：https://genmanip.axi404.top/。

英文摘要：

Robotic manipulation in real-world settings remains challenging, especially regarding robust generalization. Existing simulation platforms lack sufficient support for exploring how policies adapt to varied instructions and scenarios. Thus, they lag behind the growing interest in instruction-following foundation models like LLMs, whose adaptability is crucial yet remains underexplored in fair comparisons. To bridge this gap, we introduce GenManip, a realistic tabletop simulation platform tailored for policy generalization studies. It features an automatic pipeline via LLM-driven task-oriented scene graph to synthesize large-scale, diverse tasks using 10K annotated 3D object assets. To systematically assess generalization, we present GenManip-Bench, a benchmark of 200 scenarios refined via human-in-the-loop corrections. We evaluate two policy types: (1) modular manipulation systems integrating foundation models for perception, reasoning, and planning, and (2) end-to-end policies trained through scalable data collection. Results show that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios. We anticipate this platform to facilitate critical insights for advancing policy generalization in realistic conditions. Project Page: https://genmanip.axi404.top/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。