通过数据合成与统一规划扩展基于手册的家电操作规模
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning
- Center on Frontiers of Computing Studies, School of Computer Science, Peking University（北京大学计算机学院计算前沿研究中心）
- Beijing University of Aeronautics and Astronautics（北京航空航天大学）
- Jingdong Technology Information Technology Co., Ltd（京东科技信息技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对现有大模型缺乏支持基于手册的家电操作规划的数据集的问题，提出MAGE数据合成流水线构建UseAppliance数据集，开发AppliancePlan模型，在RealAppliance-Bench和真实机器人实验中表现优异，推进通用家用机器人发展。
AI中文摘要：
操作家用设备需要依赖状态、对干扰具有鲁棒性的长程规划，但现有大模型存在不足，因为缺乏足够多样、面向任务的数据集来支持此类规划。为弥合这一差距，我们提出MAGE，这是一种可扩展的数据合成流水线，引入了新型分层家电图（Hierarchical Appliance Graph，HAG），可从家电手册自动生成部件定位、长程规划和闭环恢复数据。借助MAGE，我们构建了UseAppliance，这是首个基于手册的家电操作规划的大规模数据集，涵盖22个家电类别，包含8.9万+个部件标注、5.3万+个操作任务以及3.3万+个闭环调整步骤。基于UseAppliance，我们开发了AppliancePlan，这是一种用于基于手册的家电操作规划的端到端模型。在RealAppliance-Bench上，仅7B参数的AppliancePlan在开环规划上实现了比最佳基线高10倍以上的性能，且在所有任务中均持续优于现有最先进模型。对6种家用设备的真实机器人实验进一步证实了有效的仿真到真实的迁移，标志着向通用家用机器人迈出了重要一步。
英文摘要：
Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10x the best baseline on open-loop planning and consistently outperforms state-of-the-art models across all tasks. Real-robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics.
