GIF：面向机器人学习的交互式与功能性物体组合的智能体生成

中文摘要 AI 辅助

机器人操作基础模型需要跨多样场景的可扩展评估和数据生成，而仿真为两者提供了环境。自动化场景生成为此提供了一条有前景的路径，然而先前的工作主要强调粗粒度的场景布局，而非细粒度的功能性物体组合。受此差距的启发，我们提出了GIF，一个用于交互式与功能性物体组合的智能体生成框架。在该框架中，我们将此问题重新表述为解耦重建后接相对位姿恢复。CoGen利用2D和3D生成模型的互补优势，生成具有粗略初始位姿的实例解耦网格。GPRM在几何与物理联合引导下优化相对位姿，并由视觉语言模型（VLM）验证器选择最符合结构化规范（structured specification）的候选。我们进一步构建了一个涵盖八类代表性接触几何类别的基准，并与最先进的生成器进行比较；GIF在资产质量和关系匹配上均有所提升，同时将碰撞率降低至1%以下。最后，我们为策略学习合成数据，揭示了在仿真和真实世界部署中的多样性扩展。

英文摘要

Robot manipulation foundation models require scalable evaluation and data generation across diverse scenarios, with simulation providing an environment for both. Automated scene generation offers a promising path, yet prior work has largely emphasized coarse-grained scene layouts rather than fine-grained functional object compositions. Motivated by this gap, we present GIF, an agentic Generation framework for Interactive and Functional object compositions. In this framework, we recast this problem as disentangled reconstruction followed by relative pose recovery. CoGen produces instance-disentangled meshes with coarse initial poses leveraging complementary strengths of 2D and 3D generative models. GPRM refines the relative pose under joint geometric and physical guidance, and a VLM verifier selects the candidate that best matches the structured specification. We further construct a benchmark spanning eight representative contact-geometry classes and compare with state-of-the-art generators; GIF improves both asset quality and relation matching, while reducing collision rate to below 1%. Finally, we synthesize data for policy learning, revealing diversity scaling in both simulation and real-world deployment.

发表机构

• Peking University（北京大学）
• Zhejiang University（浙江大学）
• The University of Hong Kong（香港大学）
• Tsinghua University（清华大学）
• Beijing Academy of Artificial Intelligence（北京人工智能研究院）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

50 pages, including supplementary material