CoDex: 无需演示学习组合式灵巧功能性操作
CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations
AI总结：
提出零演示框架CoDex，结合视觉语言模型与强化学习，自主发现并执行涉及物体内部机制操控的灵巧操作策略，在多种工具任务中验证了其有效性。
摘要：
在这项工作中，我们研究了组合式灵巧功能性物体操作（CD-FOM）：例如对植物喷洒喷雾瓶或对木材使用热胶枪等任务，这些任务既需要操作物体的内部机制，又需要控制其姿态以将物体的功能应用于环境。由于需要将物体功能、操作模式和应用区域的语义理解与精细的物理灵巧性（如抓取稳定性、运动轨迹和操作）相结合，这些任务给机器人带来了重大挑战。我们提出了CoDex，一个零演示框架，能够自主发现CD-FOM操作策略。CoDex使用视觉语言模型（VLM）从任务和场景中推断语义约束。这些约束指导分析性约束优化，生成一小组功能性抓取候选，然后通过强化学习高效优化，生成完整的抓取-移动-操作策略，并可模拟迁移到现实世界。我们在一个7自由度机器人手臂和16自由度多指手上评估了CoDex，涉及六种CD-FOM任务，包括喷雾瓶、热胶枪、空气除尘器、手电筒和胡椒研磨器等具有内部机制的未见物体，以及它们应用于未见目标物体，展示了其无需人类演示即可自主发现和执行复杂、物理可行的灵巧行为的能力。更多信息请访问此网址。
英文摘要：
In this work, we study Compositional Dexterous Functional Object Manipulation (CD-FOM): tasks such as aiming and actuating a spray bottle on a plant or a glue gun on wood, which require both actuating an object's internal mechanism and controlling its pose to apply the object's function to the environment. These tasks pose significant challenges for robots due to the demanding integration of semantic understanding of the object's function, actuation mode, and application area with intricate physical dexterity to manage grasp stability, movement trajectory, and actuation. We introduce CoDex, a zero-demonstration framework that autonomously discovers CD-FOM manipulation strategies. CoDex uses vision-language models (VLMs) to infer semantic constraints from the task and scene. These constraints guide analytic constrained optimization to generate a short list of functional grasp candidates that can be efficiently refined with reinforcement learning to generate full grasp-move-actuate policies transferable from simulation to the real world. We evaluate CoDex on a 7-DoF robot arm with a 16-DoF multi-fingered hand across six CD-FOM tasks involving previously unseen objects with internal mechanisms, including spray bottles, hot glue guns, air dusters, flashlights, and pepper grinders, and their application to unseen target objects, showcasing its ability to autonomously discover and execute complex, physically viable dexterous behaviors without human demonstrations. More information at https://robin-lab.cs.utexas.edu/CoDex/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
