通过多模态教学手册排序理解多模态程序性知识
Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals
AI总结：
本文通过构建多模态教学手册数据集，评估并揭示了现有模型在无序多模态指令排序上的不足，并提出顺序感知预训练技术，使性能提升超过5%。
摘要：
对无序事件进行排序是理解和推理现实世界任务程序的一项基本能力，这通常需要对时间常识和多模态信息有透彻的理解，因为这些程序往往通过文本和图像的组合来传达。这种能力对于顺序任务规划和多源指令摘要等应用至关重要。虽然人类能够对无序的多模态程序指令进行推理和排序，但当前的机器学习模型是否具备这种基本能力仍然是一个未决问题。在这项工作中，我们通过从流行的在线教学手册中整理数据集并收集全面的人工标注，对模型在无序多模态指令上的推理和排序能力进行了基准测试。我们发现模型不仅表现显著差于人类，而且似乎无法有效利用多模态信息。为了提高机器在多模态事件排序上的表现，我们提出了顺序感知预训练技术，利用文本和图像的顺序对齐特性，取得了超过5%的显著改进。
英文摘要：
The ability to sequence unordered events is an essential skill to comprehend and reason about real world task procedures, which often requires thorough understanding of temporal common sense and multimodal information, as these procedures are often communicated through a combination of texts and images. Such capability is essential for applications such as sequential task planning and multi-source instruction summarization. While humans are capable of reasoning about and sequencing unordered multimodal procedural instructions, whether current machine learning models have such essential capability is still an open question. In this work, we benchmark models' capability of reasoning over and sequencing unordered multimodal instructions by curating datasets from popular online instructional manuals and collecting comprehensive human annotations. We find models not only perform significantly worse than humans but also seem incapable of efficiently utilizing the multimodal information. To improve machines' performance on multimodal event sequencing, we propose sequentiality-aware pretraining techniques that exploit the sequential alignment properties of both texts and images, resulting in > 5% significant improvements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
