让心智构想，让双手实现：遵循多模态指令的长 horizon 机器人操作
Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation
AI总结：
本文提出 Orchestra 框架，将任务规划与执行管理同动作生成分离，引入 RMBench-Compose 基准，在 RMBench 任务及现实多步骤任务中均显著优于现有方法，可提升机器人长序列操作能力。
摘要：
机器人遵循多步骤指令时，需在场景变化的情况下按正确顺序作用于正确对象，还需保持对任务进度的感知并确定何时推进。为应对这些挑战，我们引入 Orchestra，这一智能体 harness（框架）将任务规划与执行管理同动作生成分离。Orchestra 将完整多模态指令分解为有序子任务，跟踪已完成步骤，并在执行过程中更新相关对象与目标的位置。它维护包含活跃子任务、空间定位及近期观测的共享上下文，动作策略利用该上下文生成动作，而单独训练的进度评估器则估计何时推进至下一个子任务。为评估在原子指令与短轨迹上训练的机器人能否执行更长的子任务序列，我们引入名为 RMBench-Compose 的基准。Orchestra 在 9 项 RMBench 任务上的平均成功率为 68.22%，较 Mem-0 高出 26.22 个百分点；在 RMBench-Compose 上，其平均成功长度（ASL）为 7.35 个子任务，是 HarnessVLA 的 2.46 倍；在 4 项现实世界多步骤任务中，其平均 ASL 较使用基础 π₀.₅ 策略的 Orchestra 提升 25.0%。这些结果表明，Orchestra 可帮助机器人执行更长的子任务序列，同时保持要求的子任务顺序并作用于指定对象。我们的代码与项目页面可在该 https URL 获取。
英文摘要：
A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $π_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
