EmRACE-3K：复杂环境中的具身推理与行动
EmbRACE-3K: Embodied Reasoning and Action in Complex Environments
AI总结：
针对VLMs在具身交互环境中空间推理与长程规划能力不足的问题，提出含3000多个任务的EmRACE-3K数据集与基准，并通过微调Qwen2.5-VL-7B结合强化学习显著提升了模型在探索、动态空间语义推理及多阶段目标执行上的具身推理能力。
摘要：
近期先进的视觉语言模型（VLMs）在被动、离线的图像和视频理解任务上表现出色。然而，它们在需要在线交互和主动场景理解的具身环境中的有效性仍然有限。在此类场景中，智能体从第一人称视角感知环境，每个动作动态地塑造后续观察。即使是最先进的模型，如GPT-4o、Claude 3.5 Sonnet和Gemini 2.5 Pro，在开放环境交互中也表现挣扎，在空间推理和长程规划方面表现出明显的局限性。为了弥补这一差距，我们引入了EmRACE-3K，这是一个包含超过3,000个语言引导任务的数据集，这些任务位于使用Unreal Engine和UnrealCV-Zoo框架构建的多样化、照片级真实环境中。任务涵盖广泛的具身挑战，包括导航、物体操作和多阶段目标执行。每个任务以多步轨迹展开，将第一人称视觉观察与高级指令、基础动作和表达智能体每步意图的自然语言理据配对。使用EmRACE-3K，我们建立了一个基准，从三个关键维度评估VLMs的具身推理能力：探索、动态空间语义推理和多阶段目标执行。在零样本设置中，所有模型的成功率均低于20%，凸显了我们基准带来的挑战以及VLMs在交互环境中的当前局限。为展示EmRACE-3K的实用性，我们进一步使用监督学习微调Qwen2.5-VL-7B，随后进行强化学习。该方法在所有三个挑战类别中均带来显著提升，突显了该数据集在赋能具身推理能力开发方面的有效性。
英文摘要：
Recent advanced vision-language models(VLMs) have demonstrated strong performance on passive, offline image and video understanding tasks. However, their effectiveness in embodied settings, which require online interaction and active scene understanding remains limited. In such scenarios, an agent perceives the environment from a first-person perspective, with each action dynamically shaping subsequent observations. Even state-of-the-art models such as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro struggle in open-environment interactions, exhibiting clear limitations in spatial reasoning and long-horizon planning. To address this gap, we introduce EmRACE-3K, a dataset of over 3,000 language-guided tasks situated in diverse, photorealistic environments constructed using Unreal Engine and the UnrealCV-Zoo framework. The tasks encompass a wide range of embodied challenges, including navigation, object manipulation, and multi-stage goal execution. Each task unfolds as a multi-step trajectory, pairing first-person visual observations with high-level instructions, grounded actions, and natural language rationales that express the agent's intent at every step. Using EmRACE-3K, we establish a benchmark to evaluate the embodied reasoning capabilities of VLMs across three key dimensions: Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution. In zero-shot settings, all models achieve success rates below 20%, underscoring the challenge posed by our benchmark and the current limitations of VLMs in interactive environments. To demonstrate the utility of EmRACE-3K, we further fine-tune Qwen2.5-VL-7B using supervised learning followed by reinforcement learning. This approach yields substantial improvements across all three challenge categories, highlighting the dataset's effectiveness in enabling the development of embodied reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
