模拟虚拟环境已被广泛用于学习执行日常家务任务的机器人智能体。这些环境极大地促进了研究进展,但通常提供有限的对象交互性、与真实环境不同的视觉外观,或相对较小的环境规模。这阻碍了虚拟场景中学习到的模型直接部署。为弥合学习环境与部署(即真实)环境之间的差距,我们提出了 ReALFRED 基准,它采用真实世界的场景、对象和房间布局,通过理解自由形式语言指令并与大型、多房间和3D捕获场景中的对象交互,来学习智能体完成家务任务。具体而言,我们扩展了 ALFRED 基准,更新了更大的环境空间并缩小了视觉域差距。借助 ReALFRED,我们分析了为 ALFRED 基准先前设计的方法,观察到它们在所有指标上表现均持续下降,鼓励社区在更真实的环境中开发方法。我们的代码和数据已公开。
英文摘要:
Simulated virtual environments have been widely used to learn robotic agents that perform daily household tasks. These environments encourage research progress by far, but often provide limited object interactability, visual appearance different from real-world environments, or relatively smaller environment sizes. This prevents the learned models in the virtual scenes from being readily deployable. To bridge the gap between these learning environments and deploying (i.e., real) environments, we propose the ReALFRED benchmark that employs real-world scenes, objects, and room layouts to learn agents to complete household tasks by understanding free-form language instructions and interacting with objects in large, multi-room and 3D-captured scenes. Specifically, we extend the ALFRED benchmark with updates for larger environmental spaces with smaller visual domain gaps. With ReALFRED, we analyze previously crafted methods for the ALFRED benchmark and observe that they consistently yield lower performance in all metrics, encouraging the community to develop methods in more realistic environments. Our code and data are publicly available.
UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);State Key Laboratory of Complex & Critical Software Environment, Beihang University(北航复杂与关键软件环境国家重点实验室);School of Computer Science, Institute for Artificial Intelligence, State Key Laboratory of General Artificial Intelligence, Peking University(北京大学计算机学院);City University of Macau(澳门城市大学);National University of Singapore(新加坡国立大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院)
Facebook AI Research(脸书人工智能研究院);Simon Fraser University(西蒙弗雷泽大学);Facebook Reality Labs(脸书现实实验室);Georgia Institute of Technology(佐治亚理工学院);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校)
我们介绍了Habitat,一个研究嵌入式人工智能(AI)的平台。Habitat能够在高效的逼真3D仿真中训练具体的代理(虚拟机器人)。具体来说,Habitat包括:(i)Mixat-Sim:一个灵活的,高性能的3D模拟器,具有可配置的代理,传感器和通用3D数据集处理。Materat-Sim速度很快--当从Matterport 3D渲染场景时,它可以在单线程运行时达到每秒数千帧(fps),并且可以在单个GPU上达到超过10,000 fps的多进程。(ii)Apriat-API:一个模块化的高级库,用于嵌入式AI算法的端到端开发-定义任务(例如,导航、指令遵循、问答)、配置、训练和基准测试具体化代理。这些大规模的工程贡献使我们能够回答科学问题,需要实验,直到现在不切实际或“只是”不切实际。具体而言,在点目标导航的上下文中:(1)我们重新审视了最近两项工作中的学习和SLAM方法之间的比较,并找到了相反结论的证据-如果将学习扩展到比以前的调查更多的经验数量级,则学习优于SLAM,以及(2)我们进行了第一次跨数据集泛化实验{训练,测试} x {Matterport 3D,Gibson},并发现只有具有深度(D)传感器的智能体才能在数据集上泛化。我们希望我们的开源平台和这些发现将推动嵌入式AI的研究。
Meta AI Research(Meta人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
像图像-目标导航这样的现实长期任务涉及探索和利用阶段。由于目标的图像,一个具体的代理必须探索发现目标,即,使用学习的先验知识进行有效搜索。一旦目标被发现,智能体必须准确地校准到目标的最后一英里的导航。与任何强大的系统一样,探索性目标发现和利用性最后一英里导航之间的切换可以更好地从错误中恢复。根据这些直观的导轨,我们提出了SLING,以提高现有的图像目标导航系统的性能。作为对现有方法的补充,我们专注于最后一英里导航,并利用神经描述符来利用问题的底层几何结构。通过简单但有效的开关,我们可以轻松地将SLING与启发式、强化学习和神经模块化策略联系起来。在标准化的图像目标导航基准(Hahn et al. 2021)上,我们提高了策略,场景和情节复杂性的性能,将最先进的成功率从45%提高到55%。除了真实感模拟,我们在三个物理场景中进行了真实的机器人实验,并发现这些改进可以很好地转移到真实环境中。
GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CFCS, School of Computer Science, Peking University(北京大学计算机科学系);Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加州大学伯克利分校);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);Galbot
RoboTidy : A 3D Gaussian Splatting Household Tidying Benchmark for Embodied Navigation and Action
Huazhong University of Science and Technology(华中科技大学);The University of Hong Kong(香港大学);INFIFORCE Intelligent Technology Co., Ltd.(INFIFORCE智能技术有限公司);Zhejiang University(浙江大学);Guangming Lab, Shenzhen(深圳光明实验室)
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Midea Group(美的集团);The Hong Kong University of Science and Technology(香港科技大学)
PR2: A Physics- and Photo-realistic Humanoid Testbed with Pilot Study in Competition
State Key Laboratory of General Artificial Intelligence(一般人工智能国家重点实验室);Beijing Institute for General Artificial Intelligence (BIGAI)(一般人工智能北京研究院);School of Future Science and Engineering(未来科学与工程学院);Soochow University(苏州大学);Department of Computer Science(计算机科学系);Harbin Institute of Technology(哈尔滨工业大学);School of Mechatronics Engineering(机械电子工程学院)
CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding
Hunan University(湖南大学);National Engineering Research Center of Robot Visual Perception and Control Technology(国家机器人视觉感知与控制技术工程研究中心);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院)
A 3D Isovist World Model -- Revealing a City's Unseen Geometry and Its Emergent Cross-City Signature
The Bartlett School of Sustainable Construction University College London, UK(可持续建设学院伦敦大学学院,英国);Department of Geography University College London, UK(地理系伦敦大学学院,英国);School of Project Management, Faculty of Engineering The University of Sydney, AU(工程学院项目管理学院悉尼大学,澳大利亚);School of Engineering Cardiff University, UK(工程学院卡迪夫大学,英国);School of Architecture Tsinghua University, Beijing, CN(建筑学院清华大学,北京,中国)
Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术研究中心);School of Computer Science and Technology, Xinjiang University(新疆大学计算机科学与技术学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);School of Electrical Engineering and Automation, Tianjin University of Technology(天津理工大学电气工程与自动化学院)