arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.07553cs.AI
收藏

VirtualEnv: 一个用于具身AI研究的平台

VirtualEnv: A Platform for Embodied AI Research

Kabir Swain, Sijie Han, Ayush Raina, Jin Zhang, Shuang Li, Michael Stopa, Antonio Torralba

AI总结:

VirtualEnv是一个基于Unreal Engine 5的开源平台,用于评估LLM在具身和交互场景中的能力,支持多智能体协作和程序生成任务,旨在推动AI与游戏的交叉研究。

摘要:

随着大语言模型(LLMs)在推理和决策能力上持续提升,对能够真实交互的环境进行严格评估的需求也日益增加。我们提出了VirtualEnv,一个基于Unreal Engine 5的下一代模拟平台,能够对LLMs在具身和交互场景中的能力进行细致的基准测试。VirtualEnv支持丰富的智能体-环境交互,包括物体操作、导航和适应性多智能体协作,以及受游戏启发的机制,如逃脱室和程序生成环境。我们提供了一个基于Unreal Engine的用户友好API,允许研究人员通过自然语言指令部署和控制LLM驱动的智能体。我们集成了大规模LLM和视觉-语言模型(VLMs),如基于GPT的模型,以从多模态输入中生成新颖的环境和结构化任务。我们的实验在逐渐增加的复杂性任务上基准测试了几种流行的LLM性能,分析了适应性、规划和多智能体协作之间的差异。我们还描述了我们的方法论,包括程序任务生成、任务验证和实时环境控制。VirtualEnv作为开源平台发布,旨在推动人工智能与游戏的交叉研究,实现LLMs在具身AI环境中的标准化评估,并为未来沉浸式模拟和互动娱乐的发展铺平道路。

英文摘要:

As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑