Structural understanding of complex visual objects is an important unsolved component of artificial intelligence. To study this, we develop a new technique for the recently proposed Break-and-Make problem in LTRON where an agent must learn to build a previously unseen LEGO assembly using a single interactive session to gather information about its components and their structure. We attack this problem by building an agent that we call \textbf{\ours} that is able to make its own visual instruction book. By disassembling an unseen assembly and periodically saving images of it, the agent is able to create a set of instructions so that it has the information necessary to rebuild it. These instructions form an explicit memory that allows the model to reason about the assembly process one step at a time, avoiding the need for long-term implicit memory. This in turn allows us to train on much larger LEGO assemblies than has been possible in the past. To demonstrate the power of this model, we release a new dataset of procedurally built LEGO vehicles that contain an average of 31 bricks each and require over one hundred steps to disassemble and reassemble. We train these models using online imitation learning which allows the model to learn from its own mistakes. Finally, we also provide some small improvements to LTRON and the Break-and-Make problem that simplify the learning environment and improve usability.
Self-Instruct: Aligning Language Models with Self-Generated Instructions
University of Washington(华盛顿大学);Tehran Polytechnic(德黑兰理工大学);Arizona State University(亚利桑那州立大学);Johns Hopkins University(约翰斯·霍普金斯大学);Allen Institute for AI(艾伦人工智能研究所)
Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
College of AI, Tsinghua University(清华大学人工智能学院);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute for AI, Tsinghua University(清华大学人工智能研究院)