STaR:可扩展的任务条件检索用于长时多模态机器人记忆
STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory
AI总结:
STaR通过任务条件检索算法提升机器人长时多模态记忆的可扩展性和上下文推理能力
摘要:
移动机器人常在多样且动态的开放场景中长时间部署,包括室内设置如仓库和制造设施,以及户外设置如农业和道路作业。核心挑战是构建一个可扩展的长时记忆,以支持代理工作流,用于在不同粒度的开放式指令上进行规划、检索和推理,并产生精确且可操作的答案。我们提出了STaR,一个代理推理框架,它(i)构建了一个任务无关的多模态长时记忆,能够泛化到未见过的查询,同时保留细粒度的环境语义(对象属性、空间关系和动态事件),(ii)引入了一种基于信息瓶颈原理的可扩展任务条件检索算法,从长时记忆中提取一个紧凑、非冗余、信息丰富的候选记忆集用于上下文推理。我们在NaVQA(混合室内/户外校园场景)和WH-VQA(一个定制的仓库基准,使用Isaac Sim构建,包含许多视觉相似的对象)上评估STaR,强调上下文推理。在两个数据集中,STaR一致优于强大的基线,实现了更高的成功率和显著更低的空间误差。我们进一步在室内和户外环境中部署STaR于真实Husky轮式机器人上,展示了鲁棒的长时推理、可扩展性和实际应用价值。项目网站:https://trailab.github.io/STaR-website/
英文摘要:
Mobile robots are often deployed over long durations in diverse open, dynamic scenes, including indoor setting such as warehouses and manufacturing facilities, and outdoor settings such as agricultural and roadway operations. A core challenge is to build a scalable long-horizon memory that supports an agentic workflow for planning, retrieval, and reasoning over open-ended instructions at variable granularity, while producing precise, actionable answers for navigation. We present STaR, an agentic reasoning framework that (i) constructs a task-agnostic, multimodal long-term memory that generalizes to unseen queries while preserving fine-grained environmental semantics (object attributes, spatial relations, and dynamic events), and (ii) introduces a Scalable Task Conditioned Retrieval algorithm based on the Information Bottleneck principle to extract from long-term memory a compact, non-redundant, information-rich set of candidate memories for contextual reasoning. We evaluate STaR on NaVQA (mixed indoor/outdoor campus scenes) and WH-VQA, a customized warehouse benchmark with many visually similar objects built with Isaac Sim, emphasizing contextual reasoning. Across the two datasets, STaR consistently outperforms strong baselines, achieving higher success rates and markedly lower spatial error. We further deploy STaR on a real Husky wheeled robot in both indoor and outdoor environments, demonstrating robust long horizon reasoning, scalability, and practical utility. Project Website: https://trailab.github.io/STaR-website/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。