嗖！Rattle！咚！--听起来不错的行动
Swoosh! Rattle! Thump! -- Actions that Sound
摘要：
真正的智能体需要捕捉所有感官的相互作用，以建立对世界的丰富物理理解。在机器人技术中，我们已经看到了视觉和触觉感知的巨大进步;然而，我们经常忽略一个关键的感觉：声音。这主要是由于缺乏捕捉动作和声音相互作用的数据。在这项工作中，我们进行了第一次大规模的研究声音和机器人动作之间的相互作用。为此，我们使用我们的机器人平台Tilt-Bot创建了最大的声音-动作-视觉数据集，其中包含60个对象上的15，000个交互。通过倾斜物体并让它们撞到机器人托盘的墙壁上，我们收集了丰富的四声道音频信息。利用这些数据，我们探索了声音和行动之间的协同作用，并提出了三个关键的见解。首先，声音指示细粒度的对象类别信息，例如，声音可以区分金属螺丝刀和金属扳手。第二，声音也包含了关于动作的因果效应的信息，也就是说，给定产生的声音，我们可以预测对物体施加了什么动作。最后，从音频嵌入导出的对象表示指示隐含的物理属性。我们证明，在以前看不见的对象上，通过交互生成的音频嵌入可以比被动视觉嵌入更好地预测前向模型24%。项目视频和数据请访问https://dhiraj100892.github.io/swoosh/
英文摘要：
Truly intelligent agents need to capture the interplay of all their senses to build a rich physical understanding of their world. In robotics, we have seen tremendous progress in using visual and tactile perception; however, we have often ignored a key sense: sound. This is primarily due to the lack of data that captures the interplay of action and sound. In this work, we perform the first large-scale study of the interactions between sound and robotic action. To do this, we create the largest available sound-action-vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot. By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information. Using this data, we explore the synergies between sound and action and present three key insights. First, sound is indicative of fine-grained object class information, e.g., sound can differentiate a metal screwdriver from a metal wrench. Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied to the object. Finally, object representations derived from audio embeddings are indicative of implicit physical properties. We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings. Project videos and data are at https://dhiraj100892.github.io/swoosh/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
