arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.18550cs.ROcs.AI
收藏

ReALFRED:照片级真实环境中的具身指令跟随基准

ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments

Seoul National University(首尔大学);Yonsei University(延世大学)

Taewoong Kim, Cheolhong Min, Byeonghwi Kim, Jinyeon Kim, Wonje Jeung, Jonghyun Choi

AI总结:

针对虚拟环境与真实环境差距导致模型难部署的问题,提出基于真实3D场景的ReALFRED基准,扩展ALFRED以缩小视觉域差距,发现现有方法在所有指标上性能均下降,推动更真实的具身指令跟随研究。

摘要:

模拟虚拟环境已被广泛用于学习执行日常家务任务的机器人智能体。这些环境极大地促进了研究进展,但通常提供有限的对象交互性、与真实环境不同的视觉外观,或相对较小的环境规模。这阻碍了虚拟场景中学习到的模型直接部署。为弥合学习环境与部署(即真实)环境之间的差距,我们提出了 ReALFRED 基准,它采用真实世界的场景、对象和房间布局,通过理解自由形式语言指令并与大型、多房间和3D捕获场景中的对象交互,来学习智能体完成家务任务。具体而言,我们扩展了 ALFRED 基准,更新了更大的环境空间并缩小了视觉域差距。借助 ReALFRED,我们分析了为 ALFRED 基准先前设计的方法,观察到它们在所有指标上表现均持续下降,鼓励社区在更真实的环境中开发方法。我们的代码和数据已公开。

英文摘要:

Simulated virtual environments have been widely used to learn robotic agents that perform daily household tasks. These environments encourage research progress by far, but often provide limited object interactability, visual appearance different from real-world environments, or relatively smaller environment sizes. This prevents the learned models in the virtual scenes from being readily deployable. To bridge the gap between these learning environments and deploying (i.e., real) environments, we propose the ReALFRED benchmark that employs real-world scenes, objects, and room layouts to learn agents to complete household tasks by understanding free-form language instructions and interacting with objects in large, multi-room and 3D-captured scenes. Specifically, we extend the ALFRED benchmark with updates for larger environmental spaces with smaller visual domain gaps. With ReALFRED, we analyze previously crafted methods for the ALFRED benchmark and observe that they consistently yield lower performance in all metrics, encouraging the community to develop methods in more realistic environments. Our code and data are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑