As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.
Facebook AI Research(脸书人工智能研究院);Simon Fraser University(西蒙弗雷泽大学);Facebook Reality Labs(脸书现实实验室);Georgia Institute of Technology(佐治亚理工学院);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校)
我们介绍了Habitat,一个研究嵌入式人工智能(AI)的平台。Habitat能够在高效的逼真3D仿真中训练具体的代理(虚拟机器人)。具体来说,Habitat包括:(i)Mixat-Sim:一个灵活的,高性能的3D模拟器,具有可配置的代理,传感器和通用3D数据集处理。Materat-Sim速度很快--当从Matterport 3D渲染场景时,它可以在单线程运行时达到每秒数千帧(fps),并且可以在单个GPU上达到超过10,000 fps的多进程。(ii)Apriat-API:一个模块化的高级库,用于嵌入式AI算法的端到端开发-定义任务(例如,导航、指令遵循、问答)、配置、训练和基准测试具体化代理。这些大规模的工程贡献使我们能够回答科学问题,需要实验,直到现在不切实际或“只是”不切实际。具体而言,在点目标导航的上下文中:(1)我们重新审视了最近两项工作中的学习和SLAM方法之间的比较,并找到了相反结论的证据-如果将学习扩展到比以前的调查更多的经验数量级,则学习优于SLAM,以及(2)我们进行了第一次跨数据集泛化实验{训练,测试} x {Matterport 3D,Gibson},并发现只有具有深度(D)传感器的智能体才能在数据集上泛化。我们希望我们的开源平台和这些发现将推动嵌入式AI的研究。
FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence
LimX Dynamics;Nankai University(南开大学);The University of Hong Kong(香港大学);University of Electronic Science and Technology of China(电子科技大学);Beijing Institute of Technology(北京理工大学);Xidian University(西安电子科技大学);Zhejiang University(浙江大学);Xi’an Jiaotong University(西安交通大学)
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
University College London(伦敦大学学院);Imperial College London(伦敦帝国学院);Huawei Noah’s Ark Lab(华为诺亚实验室);ShanghaiTech University(上海科技大学);Southern University of Science(南方科技大学)
Pipette: An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics
Key Laboratory of Smart Manufacturing in Energy Chemical Process Ministry of Education(能源化工过程智能制造国家重点实验室);Department of Computer Science and Engineering(计算机科学与工程系);Department of Laboratory Medicine(实验室医学系);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院)
AeroVerse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院航空信息研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院);Key Laboratory of Target Cognition and Application Technology(TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院航空信息研究所目标认知与应用技术重点实验室)
From Screens to Scenes: A Survey of Embodied AI in Healthcare
The Cancer Research Institute(癌症研究所);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Pennsylvania(宾夕法尼亚大学);Shanghai AI Lab(上海人工智能实验室);Central South University(中南大学);National University of Singapore(新加坡国立大学);Westlake University(西湖大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
6G Communication Networks Enabling Embodied Agents: Architecture and Prototype
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Institute of Intelligent Networks and Communications (NINE), Nanjing University (Suzhou Campus)(南京大学智能网络与通信研究所(苏州校区))
Georgia Tech(佐治亚理工学院);Facebook AI Research(Facebook人工智能研究院);Simon Fraser University(西蒙菲莎大学);Imperial College London(帝国理工学院);Princeton University(普林斯顿大学);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校);Google(谷歌);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);UC San Diego(加州大学圣地亚哥分校)
ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
University of California San Diego(加州大学圣地亚哥分校);Carnegie Mellon University(卡内基梅隆大学);Hillbot(Hillbot公司);TU Dresden(德累斯顿技术大学);Tsinghua University(清华大学);King’s College London(伦敦国王学院)
BioVLN: A Simulation Platform for Visual Language Navigation in Biomedical Laboratories
East China University of Science and Technology(华东理工大学);Ruijin Hospital, Shanghai Jiao Tong University School of Medicine(上海交通大学医学院附属瑞金医院);Shanghai AI Laboratory(上海人工智能实验室)