Embodied manipulation is a fundamental ability in the realm of embodied artificial intelligence. Although current embodied manipulation models show certain generalizations in specific settings, they struggle in new environments and tasks due to the complexity and diversity of real-world scenarios. The traditional end-to-end data collection and training manner leads to significant data demands. Decomposing end-to-end tasks into atomic skills helps reduce data requirements and improves the task success rate. However, existing methods are limited by predefined skill sets that cannot be dynamically updated. To address the issue, we introduce a three-wheeled data-driven method to build an atomic skill library. We divide tasks into subtasks using the Vision-Language-Planning (VLP). Then, atomic skill definitions are formed by abstracting the subtasks. Finally, an atomic skill library is constructed via data collection and Vision-Language-Action (VLA) fine-tuning. As the atomic skill library expands dynamically with the three-wheel update strategy, the range of tasks it can cover grows naturally. In this way, our method shifts focus from end-to-end tasks to atomic skills, significantly reducing data costs while maintaining high performance and enabling efficient adaptation to new tasks. Extensive experiments in real-world settings demonstrate the effectiveness and efficiency of our approach.
Embodied Skill Bench: Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Sun Yat-Sen University(中山大学);Jilin University(吉林大学);Beijing University of Aeronautics and Astronautics(北京航空航天大学)
SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Automation, Tsinghua University(清华大学自动化系);Nanyang Technological University(南洋理工大学);Beijing Normal University(北京师范大学)
Embodied Intelligence for Flexible Manufacturing: A Survey
National University of Defense Technology(中国人民解放军国防科技大学);Wuhan University(武汉大学);Shenzhen University(深圳大学);Hunan University(湖南大学);Shenyang Institute of Automation Chinese Academy of Sciences(中国科学院沈阳自动化研究所)
RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)
EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Cornell University(康奈尔大学);Universal Ubiquitous AI Co., Ltd.(Universal Ubiquitous AI有限公司);Hangzhou DEEP Robotics Technology Co., Ltd.(杭州深度机器人科技有限公司);National University of Singapore(新加坡国立大学)
UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agents
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beihang University(北京航空航天大学);Shanxi Datong University(山西大同大学);Johns Hopkins University(约翰斯·霍普金斯大学);University of Sanya(三亚学院);Wenzhou-Kean University(温州肯恩大学);Macau University of Science and Technology(澳门科技大学);KOKONI 3D, Moxin Technology(魔芯科技KOKONI 3D);University of Science and Technology of China(中国科学技术大学)
EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学信息科学与工程学院);School of Mechanical Science and Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);Lehigh University(莱斯大学);College of AI, Tsinghua University(清华大学人工智能学院)
StateSkill-Agent: An Effective and Efficient Embodied Agent via State-conditioned Skill
The Hong Kong Polytechnic University(香港理工大学);ACE Robotics(ACE机器人公司);Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);Fudan University(复旦大学)
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院(BIGAI))
Facebook AI Research(脸书人工智能研究院);Simon Fraser University(西蒙弗雷泽大学);Facebook Reality Labs(脸书现实实验室);Georgia Institute of Technology(佐治亚理工学院);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校)
我们介绍了Habitat,一个研究嵌入式人工智能(AI)的平台。Habitat能够在高效的逼真3D仿真中训练具体的代理(虚拟机器人)。具体来说,Habitat包括:(i)Mixat-Sim:一个灵活的,高性能的3D模拟器,具有可配置的代理,传感器和通用3D数据集处理。Materat-Sim速度很快--当从Matterport 3D渲染场景时,它可以在单线程运行时达到每秒数千帧(fps),并且可以在单个GPU上达到超过10,000 fps的多进程。(ii)Apriat-API:一个模块化的高级库,用于嵌入式AI算法的端到端开发-定义任务(例如,导航、指令遵循、问答)、配置、训练和基准测试具体化代理。这些大规模的工程贡献使我们能够回答科学问题,需要实验,直到现在不切实际或“只是”不切实际。具体而言,在点目标导航的上下文中:(1)我们重新审视了最近两项工作中的学习和SLAM方法之间的比较,并找到了相反结论的证据-如果将学习扩展到比以前的调查更多的经验数量级,则学习优于SLAM,以及(2)我们进行了第一次跨数据集泛化实验{训练,测试} x {Matterport 3D,Gibson},并发现只有具有深度(D)传感器的智能体才能在数据集上泛化。我们希望我们的开源平台和这些发现将推动嵌入式AI的研究。
Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
Northeastern University, Boston, MA, USA;The Chinese University of Hong Kong, Hong Kong, China;Peking University, Beijing, China;Westlake University, Hangzhou, China;Harvard University, Cambridge, MA, USA;Purdue University, West Lafayette, IN, USA;University of Oxford, Oxford, United Kingdom
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University(里士大学);U.S. National Institute of Standards and Technology(美国国家标准与技术研究院);Massachusetts Institute of Technology(麻省理工学院);Karlsruhe Institute of Technology(卡尔斯鲁厄技术大学);Autodesk Research(Autodesk研究);University of California, Berkeley(加州大学伯克利分校);KTH Royal Institute of Technology(皇家理工学院);Tsinghua University(清华大学);Columbia University(哥伦比亚大学);ASTM International(美国材料与试验协会);Carnegie Mellon University(卡内基梅隆大学);German Aerospace Center (DLR)(德国航空航天中心);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Texas at Austin(德克萨斯大学奥斯汀分校);NVIDIA Research(NVIDIA研究)
Never-Ending Behavior-Cloning Agent for Robotic Manipulation
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);School of Automation Science and Engineering, South China University of Technology(自动化科学与工程学院,华南理工大学);Mohamed bin Zayed University of Artificial Intelligence(马斯达尔大学人工智能学院)
SpaceMind: A Modular and Self-Evolving Embodied Vision-Language Agent Framework for Autonomous On-orbit Servicing
University of Chinese Academy of Sciences(中国科学院大学);Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间利用技术与工程中心)
Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China(人工智能学院,智能与计算学院,天津大学,中国);School of Computer Science and Information Engineering, Hefei University of Technology, China(计算机科学与信息工程学院,合肥工业大学,中国)
本文提出Decompose and Recompose框架,通过分解现有任务演示为可解释的技能-动作对,实现跨任务机械操作的零样本泛化。