The domain of Embodied AI, in which agents learn to complete tasks through interaction with their environment from egocentric observations, has experienced substantial growth with the advent of deep reinforcement learning and increased interest from the computer vision, NLP, and robotics communities. This growth has been facilitated by the creation of a large number of simulated environments (such as AI2-THOR, Habitat and CARLA), tasks (like point navigation, instruction following, and embodied question answering), and associated leaderboards. While this diversity has been beneficial and organic, it has also fragmented the community: a huge amount of effort is required to do something as simple as taking a model trained in one environment and testing it in another. This discourages good science. We introduce AllenAct, a modular and flexible learning framework designed with a focus on the unique requirements of Embodied AI research. AllenAct provides first-class support for a growing collection of embodied environments, tasks and algorithms, provides reproductions of state-of-the-art models and includes extensive documentation, tutorials, start-up code, and pre-trained models. We hope that our framework makes Embodied AI more accessible and encourages new researchers to join this exciting area. The framework can be accessed at: https://allenact.org/
A Survey of Embodied AI: From Simulators to Research Tasks
Nanyang Technological University(南洋理工大学);Singapore University of Technology and Design(新加坡科技设计大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院)
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室);Peng Cheng Laboratory(鹏城实验室);Institute of Digital Media, Peking University(北京大学数字媒体研究院)
Facebook AI Research(脸书人工智能研究院);Simon Fraser University(西蒙弗雷泽大学);Facebook Reality Labs(脸书现实实验室);Georgia Institute of Technology(佐治亚理工学院);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校)
我们介绍了Habitat,一个研究嵌入式人工智能(AI)的平台。Habitat能够在高效的逼真3D仿真中训练具体的代理(虚拟机器人)。具体来说,Habitat包括:(i)Mixat-Sim:一个灵活的,高性能的3D模拟器,具有可配置的代理,传感器和通用3D数据集处理。Materat-Sim速度很快--当从Matterport 3D渲染场景时,它可以在单线程运行时达到每秒数千帧(fps),并且可以在单个GPU上达到超过10,000 fps的多进程。(ii)Apriat-API:一个模块化的高级库,用于嵌入式AI算法的端到端开发-定义任务(例如,导航、指令遵循、问答)、配置、训练和基准测试具体化代理。这些大规模的工程贡献使我们能够回答科学问题,需要实验,直到现在不切实际或“只是”不切实际。具体而言,在点目标导航的上下文中:(1)我们重新审视了最近两项工作中的学习和SLAM方法之间的比较,并找到了相反结论的证据-如果将学习扩展到比以前的调查更多的经验数量级,则学习优于SLAM,以及(2)我们进行了第一次跨数据集泛化实验{训练,测试} x {Matterport 3D,Gibson},并发现只有具有深度(D)传感器的智能体才能在数据集上泛化。我们希望我们的开源平台和这些发现将推动嵌入式AI的研究。
EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学信息科学与工程学院);School of Mechanical Science and Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);Lehigh University(莱斯大学);College of AI, Tsinghua University(清华大学人工智能学院)
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院(BIGAI))
Australian National University(澳大利亚国立大学);Princeton University(普林斯顿大学);Carnegie Mellon University(卡内基梅隆大学);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校);George Mason University(乔治梅森大学);Facebook(脸书);Allen Institute for AI(艾伦人工智能研究所)
Georgia Tech(佐治亚理工学院);Facebook AI Research(Facebook人工智能研究院);Simon Fraser University(西蒙菲莎大学);Imperial College London(帝国理工学院);Princeton University(普林斯顿大学);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校);Google(谷歌);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);UC San Diego(加州大学圣地亚哥分校)
Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI
Tianjin University(天津大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Sun Yat-sen University(中山大学);PengCheng Laboratory(鹏城实验室);Tongji University(同济大学);University College London(伦敦大学学院);Peking University(北京大学);Tsinghua University(清华大学);Imperial College London(伦敦帝国学院);King’s College London(伦敦国王学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Manchester(曼彻斯特大学);Nanjing University(南京大学);TU Darmstadt(图宾根大学)
VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
Adelaide University(阿德莱德大学);Responsible AI Research Centre, Australian Institute for Machine Learning(负责任人工智能研究中心、澳大利亚机器学习研究所);Tongji University(同济大学);ManyCore;Zhejiang University(浙江大学);Shanghai AI Lab(上海人工智能实验室);CSIRO Data61
Multi-agent Embodied AI: Advances and Future Directions
National Key Lab of Autonomous Intelligent Unmanned Systems(国家级自主智能无人系统实验室);Beijing Institute of Technology(北京理工大学);National Key Laboratory for Novel Software Technology(新型软件技术国家实验室);Nanjing University(南京大学);State Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Xi’an Jiaotong University(西安交通大学);College of Electrical Engineering(电子工程学院);Zhejiang University(浙江大学);Clean Energy Automotive Engineering Center(清洁能源汽车工程中心);Tongji University(同济大学);Intelligent Transportation Thrust(智能交通方向);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);Georgia Tech(佐治亚理工学院);Meta AI;Carnegie Mellon University(卡内基梅隆大学);FBK(布鲁诺·凯塞勒基金会);University of Padova(帕多瓦大学);Simon Fraser University(西蒙弗雷泽大学);UT Austin(得克萨斯大学奥斯汀分校);NVIDIA(英伟达);Apple(苹果公司);Stanford University(斯坦福大学);Google(谷歌);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);UMass Amherst(马萨诸塞大学阿默斯特分校);QUT Centre for Robotics(昆士兰科技大学机器人中心);Oregon State University(俄勒冈州立大学);Intel Labs(英特尔研究院)
Resilience Meets Autonomy: Governing Embodied AI in Critical Infrastructure
Department of Automation and Process Engineering (IAP)(自动化与过程工程系);UiT The Arctic University of Norway(北极大学);Department of Technology and Safety (ITS)(技术与安全系)
The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models
College of Information Science and Technology, Jinan University(信息科学与技术学院,暨南大学);Faculty of Data Science, City University of Macau(数据科学学院,澳门城市大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Zhongguancun Laboratory(中关村实验室);College of Cyber Security, Jinan University(网络安全学院,暨南大学)
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室)
Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Action
School of Information Science and Engineering, Ocean University of China(中国海洋大学信息科学与工程学院);Innovation School of Artificial Intelligence, Hefei University of Technology(合肥工业大学人工智能创新学院);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院);ReLER Laboratory, CCAI, Zhejiang University(浙江大学计算机辅助设计与图形学国家重点实验室)
EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能国家重点实验室);University of Hong Kong(香港大学)
SocialAI 0.1: Towards a Benchmark to Stimulate Research on Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
Inria(法国国家信息与自动化研究所);Microsoft Research(微软研究院)
构建能够参与与人类的社会互动的嵌入式自主智能体是人工智能的主要挑战之一。这个问题激发了许多关于具身语言使用的研究方向。目前的方法集中在语言作为一种沟通工具,在非常简单和非多样化的社会情况下:语言的“自然性”被降低到高词汇量和可变性的概念。在本文中,我们认为,要实现人类水平的人工智能,需要更广泛的关键社交技能:1)在复杂多变的社会环境中使用语言; 2)在不断发展的社会世界中,在多模态环境中进行复杂的具体沟通。在这项工作中,我们解释了认知科学的概念如何帮助人工智能绘制通往类人智能的路线图,重点是其社会维度。然后,我们研究了最近SOTA Deep RL方法在即将推出的SocialAI的第一个网格世界环境中进行测试时的局限性,SocialAI是评估Deep RL代理社交技能的基准。视频和代码可在https://sites.google.com/view/socialai01上获得。
The Essential Role of Causality in Foundation World Models for Embodied AI
Microsoft Research(微软研究院);Technical University of Munich(慕尼黑工业大学);KTH royal institute of technology(皇家理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院)
Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);City University of Hong Kong(香港城市大学);Jilin University(吉林大学);Singapore Management University(新加坡管理大学);Deakin University(德肯大学);Tongji University(同济大学);Nanyang Technological University(南洋理工大学);Chinese Academy of Sciences(中国科学院);The University of Melbourne(墨尔本大学);Johns Hopkins University(约翰霍普金斯大学)