J-ORA：面向机器人感知的日语物体识别、指代与动作预测框架及多模态数据集
J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perception
AI总结：
J-ORA提出一个带详细物体属性标注的日语多模态数据集，支持物体识别、指代消解和动作预测，实验证明属性标注显著提升VLM感知性能。
摘要：
我们提出了J-ORA，一个新颖的多模态数据集，通过在日语人机对话场景中提供详细的物体属性标注，弥合了机器人感知领域的空白。J-ORA旨在支持三个关键的感知任务：物体识别、指代消解和下一动作预测，利用全面的属性模板（如类别、颜色、形状、大小、材质和空间关系）。对专有和开源视觉语言模型（VLMs）的广泛评估表明，与不包含物体属性相比，纳入详细物体属性显著提升了多模态感知性能。尽管有所改进，我们发现专有和开源VLM之间仍存在差距。此外，我们对物体可供性的分析显示，不同VLM在理解物体功能和上下文关系方面的能力各不相同。这些发现强调了丰富的、上下文敏感的属性标注在推动动态环境中机器人感知的重要性。项目页面见https://jatuhurrra.github.io/J-ORA/。
英文摘要：
We introduce J-ORA, a novel multimodal dataset that bridges the gap in robot perception by providing detailed object attribute annotations within Japanese human-robot dialogue scenarios. J-ORA is designed to support three critical perception tasks, object identification, reference resolution, and next-action prediction, by leveraging a comprehensive template of attributes (e.g., category, color, shape, size, material, and spatial relations). Extensive evaluations with both proprietary and open-source Vision Language Models (VLMs) reveal that incorporating detailed object attributes substantially improves multimodal perception performance compared to without object attributes. Despite the improvement, we find that there still exists a gap between proprietary and open-source VLMs. In addition, our analysis of object affordances demonstrates varying abilities in understanding object functionality and contextual relationships across different VLMs. These findings underscore the importance of rich, context-sensitive attribute annotations in advancing robot perception in dynamic environments. See project page at https://jatuhurrra.github.io/J-ORA/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
