EgoActor: 通过视觉语言模型将任务规划接地于空间感知的自我中心动作以实现人形机器人
EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models
AI总结：
EgoActor通过视觉语言模型将高层任务指令转化为精确的空间感知动作，实现人形机器人在真实和模拟环境中的稳健任务规划与运动执行。
摘要：
在真实世界环境中部署人形机器人本质上极具挑战性，因为它要求在部分信息观察和动态变化的环境中紧密集成感知、运动和操作。此外，还需要在不同类型子任务之间实现稳健的过渡。为了解决这些挑战，我们提出了一种新的任务——EgoActing，它要求将高层指令直接接地于各种精确的空间感知人形动作。我们进一步通过引入EgoActor，一种统一且可扩展的视觉语言模型（VLM），来实例化这一任务。该模型能够预测运动原始动作（例如行走、转身、侧移、改变高度）、头部运动、操作指令以及人与机器人交互，以协调感知和执行。我们利用广泛监督的自我中心RGB-only数据、空间推理问答和模拟环境演示，使EgoActor能够做出稳健、上下文感知的决策，并在1秒内执行流畅的动作推断，无论是8B还是4B参数模型。在模拟和真实世界环境中的广泛评估表明，EgoActor有效连接了抽象的任务规划和具体的运动执行，同时在多样化的任务和未见过的环境中实现泛化。
英文摘要：
Deploying humanoid robots in real-world settings is fundamentally challenging, as it demands tight integration of perception, locomotion, and manipulation under partial-information observations and dynamically changing environments. As well as transitioning robustly between sub-tasks of different types. Towards addressing these challenges, we propose a novel task - EgoActing, which requires directly grounding high-level instructions into various, precise, spatially aware humanoid actions. We further instantiate this task by introducing EgoActor, a unified and scalable vision-language model (VLM) that can predict locomotion primitives (e.g., walk, turn, move sideways, change height), head movements, manipulation commands, and human-robot interactions to coordinate perception and execution in real-time. We leverage broad supervision over egocentric RGB-only data from real-world demonstrations, spatial reasoning question-answering, and simulated environment demonstrations, enabling EgoActor to make robust, context-aware decisions and perform fluent action inference (under 1s) with both 8B and 4B parameter models. Extensive evaluations in both simulated and real-world environments demonstrate that EgoActor effectively bridges abstract task planning and concrete motor execution, while generalizing across diverse tasks and unseen environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
