arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01111cs.AIcs.RO
收藏

通过构建自己的说明书来学习搭建

Learning to Build by Building Your Own Instructions

University of Washington(华盛顿大学);NVIDIA(英伟达)

Aaron Walsman, Muru Zhang, Adam Fishman, Ali Farhadi, Dieter Fox

AI总结:

针对 LTRON 的 Break-and-Make 任务,提出可自行生成视觉说明书的 \ours 智能体,并用在线模仿学习与新 LEGO 车辆数据集验证其大规模重建能力。

摘要:

对复杂视觉对象的结构理解是人工智能中一个尚未解决的重要组成部分。为研究这一点,我们针对近期提出的 LTRON 中的 Break-and-Make 问题开发了一项新技术;在该问题中,智能体必须利用单次交互式会话来收集此前未见的 LEGO 组件及其结构的信息,并学习搭建该组件。我们通过构建一个称为 \textbf{\ours} 的智能体来攻克此问题,该智能体能够制作自己的视觉说明书。通过拆解一个未见组件并定期保存其图像,智能体能够创建一组说明书,从而掌握重建该组件所需的信息。这些说明书构成显式记忆,使模型能够一次一步地推理组装过程,避免了对长期隐式记忆的需求。这反过来使我们能够在比以往可行规模大得多的 LEGO 组件上进行训练。为展示该模型的能力,我们发布了一个由程序化生成的 LEGO 车辆组成的新数据集,每辆车平均包含 31 块积木,需要一百多个步骤才能完成拆解和重新组装。我们使用在线模仿学习训练这些模型,使模型能够从自身错误中学习。最后,我们还对 LTRON 和 Break-and-Make 问题进行了一些小幅改进,以简化学习环境并提升可用性。

英文摘要:

Structural understanding of complex visual objects is an important unsolved component of artificial intelligence. To study this, we develop a new technique for the recently proposed Break-and-Make problem in LTRON where an agent must learn to build a previously unseen LEGO assembly using a single interactive session to gather information about its components and their structure. We attack this problem by building an agent that we call \textbf{\ours} that is able to make its own visual instruction book. By disassembling an unseen assembly and periodically saving images of it, the agent is able to create a set of instructions so that it has the information necessary to rebuild it. These instructions form an explicit memory that allows the model to reason about the assembly process one step at a time, avoiding the need for long-term implicit memory. This in turn allows us to train on much larger LEGO assemblies than has been possible in the past. To demonstrate the power of this model, we release a new dataset of procedurally built LEGO vehicles that contain an average of 31 bricks each and require over one hundred steps to disassemble and reassemble. We train these models using online imitation learning which allows the model to learn from its own mistakes. Finally, we also provide some small improvements to LTRON and the Break-and-Make problem that simplify the learning environment and improve usability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑