RoboStream: 用记忆融合时空推理提升视觉语言模型在机器人中的应用
RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
AI总结:
RoboStream通过时空融合令牌和因果时空图实现持久记忆,解决机器人长时间任务中的几何锚定和状态追踪问题,提升长期任务表现。
摘要:
使可靠的长周期机器人操作成为开放世界具身智能的关键步骤。然而,基于视觉语言模型的规划器将每一步视为孤立的观测到动作映射,迫使它们在每个决策点从原始像素重新推断场景几何,同时缺乏对先前动作如何改变环境的认识。尽管在短周期表现强劲,这些系统缺乏持续的时空推理能力,无法实现持久的几何锚定和动作触发的状态转换记忆。没有持续的状态追踪,感知误差会在执行周期内累积,被暂时遮挡的对象会被灾难性遗忘,这些叠加的失败会导致先决条件违反,进而影响后续步骤。相比之下,人类保持持续的内心模型,持续跟踪空间关系和动作后果,而不是在每个瞬间重新构建。受这种人类因果时空推理能力的启发,我们提出了RoboStream,一种无需训练的框架,通过时空融合令牌实现几何锚定,将视觉证据绑定到3D几何属性以实现持久的对象定位,并通过因果时空图记录动作触发的状态转换。这种设计使规划器能够在不额外训练或微调的情况下追踪因果链并保留遮挡下的对象永久性。RoboStream在长周期RLBench上达到90.5%,在具有挑战性的现实世界积木搭建任务中达到44.4%,而SoFar和VoxPoser分别得分为11.1%,证明了时空推理和因果记忆是可靠长周期操作的关键缺失组件。
英文摘要:
Enabling reliable long-horizon robotic manipulation is a crucial step toward open-world embodied intelligence. However, VLM-based planners treat each step as an isolated observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision point while remaining unaware of how prior actions have reshaped the environment. Despite strong short-horizon performance, these systems lack the spatio-temporal reasoning required for persistent geometric anchoring and memory of action-triggered state transitions. Without persistent state tracking, perceptual errors accumulate across the execution horizon, temporarily occluded objects are catastrophically forgotten, and these compounding failures lead to precondition violations that cascade through subsequent steps. In contrast, humans maintain a persistent mental model that continuously tracks spatial relations and action consequences across interactions rather than reconstructing them at each instant. Inspired by this human capacity for causal spatio-temporal reasoning with persistent memory, we propose RoboStream, a training-free framework that achieves geometric anchoring through Spatio-Temporal Fusion Tokens (STF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding, and maintains causal continuity via a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps. This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning. RoboStream achieves 90.5% on long-horizon RLBench and 44.4% on challenging real-world block-building tasks, where both SoFar and VoxPoser score 11.1%, demonstrating that spatio-temporal reasoning and causal memory are critical missing components for reliable long-horizon manipulation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。