从指令到协助：一个对齐指令手册与装配视频的数据集用于评估多模态大语言模型
From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs
AI总结：
本文提出M2AD数据集，用于评估多模态大语言模型在技术任务中的协助能力，探讨其推理、步骤跟踪和指令引用能力。
摘要：
近年来，大型语言模型（LLMs）的进展改变了人工智能支持复杂现实任务的方式，推动研究突破文本边界，向多模态领域发展，催生了多模态大语言模型（MLMs）。鉴于当前LLM基于助手在解决技术或领域特定问题中的应用，自然趋势是扩展这些助手的输入领域，利用MLMs。理想情况下，这些MLMs应作为实时助手在工序任务中使用，希望整合用户被协助者的环境视角，甚至通过虚拟现实（VR）或增强现实（AR）支持共享同一视角，以在同一场景中推理。本文旨在评估当前公开可用的MLMs质量，以提供此类技术协助。为此，我们标注了一个带有逐步标签和手册参考的家具装配数据集：Manual to Action Dataset（M2AD）。我们使用此数据集评估（1）MLMs的推理能力能否减少对详细标注的依赖，从而实现更高效、低成本的标注实践；（2）MLMs能否跟踪装配步骤的进展；（3）MLMs能否正确引用指令手册页面。我们的结果表明，尽管某些模型理解了工序序列，但其性能受限于架构和硬件约束，凸显了多图像和交错文本图像推理的必要性。
英文摘要：
The recent advancements introduced by Large Language Models (LLMs) have transformed how Artificial Intelligence (AI) can support complex, real world tasks, pushing research outside the text boundaries towards multi modal contexts and leading to Multimodal Large Language Models (MLMs). Given the current adoption of LLM based assistants in solving technical or domain specific problems, the natural continuation of this trend is to extend the input domains of these assistants exploiting MLMs. Ideally, these MLMs should be used as real time assistants in procedural tasks, hopefully integrating a view of the environment where the user being assisted is, or even better sharing the same point of view via Virtual Reality (VR) or Augmented Reality (AR) supports, to reason over the same scenario the user is experiencing. With this work, we aim at evaluating the quality of currently openly available MLMs to provide this kind of assistance on technical tasks. To this end, we annotated a data set of furniture assembly with step by step labels and manual references: the Manual to Action Dataset (M2AD). We used this dataset to assess (1) to which extent the reasoning abilities of MLMs can be used to reduce the need for detailed labelling, allowing for more efficient, cost effective annotation practices, (2) whether MLMs are able to track the progression of assembly steps (3) and whether MLMs can refer correctly to the instruction manual pages. Our results showed that while some models understand procedural sequences, their performance is limited by architectural and hardware constraints, highlighting the need for multi image and interleaved text image reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
