语义可感知的多模态检索：基于具身记忆的层次化移动操作
Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
AI总结：
本研究提出Affordance RAG框架，通过构建具有可操作性的具身记忆，提升机器人在开放词汇移动操作中的检索性能和任务成功率。
摘要：
在本研究中，我们解决了开放词汇移动操作的问题，即机器人需要根据自由形式的自然语言指令携带各种物体到存储处。这项任务具有挑战性，因为它涉及理解视觉语义和操作动作的可操作性。为了解决这些挑战，我们提出了Affordance RAG，一种零样本层次化多模态检索框架，该框架从预探索图像中构建了具有可操作性的具身记忆。该模型根据区域和视觉语义检索候选目标，并通过可操作性分数重新排序，使机器人能够识别在现实环境中可执行的操作选项。我们的方法在大规模室内环境中移动操作指令的检索性能上优于现有方法。此外，在现实世界实验中，机器人根据自由形式指令在室内环境中执行移动操作时，所提出的方法实现了85%的任务成功率，优于现有方法在检索性能和整体任务成功率方面。
英文摘要：
In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the robot performed mobile manipulation in indoor environments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both retrieval performance and overall task success.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
