DM2RM：基于开放词汇指令的目标物体与容器双模式多模态排序
DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructions
AI总结：
针对家用服务机器人开放词汇指令下难同时识别目标物体与容器的问题，提出DM2RM双模式多模态排序模型，通过模式切换机制实现单模型双目标检索，在新数据集和真实机器人平台上性能优于现有方法，零样本任务成功率达82%。
摘要：
本研究旨在开发一款家用服务机器人（DSR），该机器人可在开放词汇指令的引导下，将日常物品搬运至指定的家具处。现有方法中，很少有能在图像检索场景下处理带开放词汇指令的移动操作任务，且多数方法无法同时识别目标物体和容器。我们提出了双模式多模态排序模型（DM2RM），该模型基于多模态基础模型，可通过单个模型同时检索目标物体和容器的图像。我们引入了一种切换机制，利用模式标记（mode token）和大语言模型实现的短语识别功能，根据预测目标切换嵌入空间。为评估DM2RM，我们构建了一个全新数据集，其中包含从数百个建筑规模环境中采集的真实世界图像，以及带指代表达的众包指令。评估结果表明，在图像检索场景下，所提出的DM2RM在标准指标上优于现有方法。此外，我们在包含取放搬运动作的标准化真实世界DSR平台上验证了DM2RM的应用，尽管处于零样本迁移设置下，其任务成功率仍达到82%。演示视频、代码及更多材料可访问https://kkrr10.github.io/dm2rm/获取。
英文摘要：
In this study, we aim to develop a domestic service robot (DSR) that, guided by open-vocabulary instructions, can carry everyday objects to the specified pieces of furniture. Few existing methods handle mobile manipulation tasks with open-vocabulary instructions in the image retrieval setting, and most do not identify both the target objects and the receptacles. We propose the Dual-Mode Multimodal Ranking model (DM2RM), which enables images of both the target objects and receptacles to be retrieved using a single model based on multimodal foundation models. We introduce a switching mechanism that leverages a mode token and phrase identification via a large language model to switch the embedding space based on the prediction target. To evaluate the DM2RM, we construct a novel dataset including real-world images collected from hundreds of building-scale environments and crowd-sourced instructions with referring expressions. The evaluation results show that the proposed DM2RM outperforms previous approaches in terms of standard metrics in image retrieval settings. Furthermore, we demonstrate the application of the DM2RM on a standardized real-world DSR platform including fetch-and-carry actions, where it achieves a task success rate of 82% despite the zero-shot transfer setting. Demonstration videos, code, and more materials are available at https://kkrr10.github.io/dm2rm/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
