ProMQA-Assembly：多模态装配任务问答数据集
ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
AI总结：
本文提出ProMQA-Assembly数据集，包含646个多模态问答对，用于评估装配任务中的人机交互系统，通过半自动化标注方法生成问题并结合细粒度动作标签提升多样性，验证了推理模型在复杂多模态任务中的表现。
摘要：
在装配任务中，助手能够帮助人类完成日常任务到工业场景的交互。然而，装配活动的评估资源仍显不足。为促进系统开发，我们提出了一个新的多模态问答评估数据集ProMQA-Assembly。该数据集包含646个问答对，要求在在线式方式下对人类活动视频及其指令手册进行多模态理解。为提高数据创建的效率，我们采用半自动化问答标注方法，其中LLMs生成候选问答对，人类进行验证。我们进一步通过整合细粒度动作标签来多样化问题类型。此外，我们创建了81个指令任务图用于目标装配任务。这些新创建的任务图被用于我们的基准测试实验，以及促进人类验证过程。通过我们的数据集，我们基准测试了包括竞争性专有多模态模型在内的模型。我们发现ProMQA-Assembly包含具有挑战性的多模态问题，其中推理模型展示了有希望的结果。我们相信我们的新评估数据集有助于进一步发展流程活动助手。
英文摘要：
Assistants on assembly tasks show great potential to benefit humans ranging from helping with everyday tasks to interacting in industrial settings. However, evaluation resources in assembly activities are underexplored. To foster system development, we propose a new multimodal QA evaluation dataset on assembly activities. Our dataset, ProMQA-Assembly, consists of 646 QA pairs that require multimodal understanding of human activity videos and their instruction manuals in an online-style manner. For cost effectiveness in the data creation, we adopt a semi-automated QA annotation approach, where LLMs generate candidate QA pairs and humans verify them. We further improve QA generation by integrating fine-grained action labels to diversify question types. Additionally, we create 81 instruction task graphs for our target assembly tasks. These newly created task graphs are used in our benchmarking experiment, as well as in facilitating the human verification process. With our dataset, we benchmark models, including competitive proprietary multimodal models. We find that ProMQA-Assembly contains challenging multimodal questions, where reasoning models showcase promising results. We believe our new evaluation dataset contributes to the further development of procedural-activity assistants.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
