arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17022cs.RO

基于多图像及自动指标增强的移动操作指令生成

Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement

Keio University（庆应义塾大学）

Kei Katsumata, Motonari Kambara, Daichi Yashima, Ryosuke Korekata, Komei Sugiura

备注：
Accepted for IEEE RA-L 2025
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对单图像模型无法生成双图像移动操作指令的问题，提出处理目标物体与容器图像的模型，并引入结合学习与n-gram评估指标作奖励的训练方法，在自动指标上超越基线，物理实验证明其数据增强能提升现有模型性能。

摘要：

我们考虑基于目标物体图像和容器图像生成自由形式移动操作指令的问题。传统图像描述模型无法生成合适的指令，因为其架构通常针对单图像进行优化。在本研究中，我们提出了一种同时处理目标物体和容器以生成移动操作任务自由形式指令句子的模型。此外，我们引入了一种新颖的训练方法，该方法将基于学习和基于 n-gram 的自动评估指标的分数作为奖励进行有效结合。该方法使模型能够学习单词之间的共现关系和适当的释义。结果表明，我们提出的方法在标准自动评估指标上优于包括代表性多模态大语言模型在内的基线方法。此外，物理实验表明，使用我们的方法对语言指令进行数据增强，可提高现有用于移动操作的多模态语言理解模型的性能。

英文摘要：

We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that handles both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Moreover, we introduce a novel training method that effectively incorporates the scores from both learning-based and n-gram based automatic evaluation metrics as rewards. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. Results demonstrate that our proposed method outperforms baseline methods including representative multimodal large language models on standard automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑