InsTALL：基于多模态大语言模型的上下文感知指令任务辅助
InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Models
AI总结：
InsTALL利用多模态大语言模型和自动提取的任务图，通过在线视觉流实时辅助用户执行多步骤任务，在任务识别、动作识别、动作预测和计划预测等子任务上达到最先进性能，并优于错误识别基线。
摘要：
生成模型能力的提升有助于构建利用语言之外模态的多模态虚拟助手。通过观察人类执行多步骤任务，可以构建对正在执行的动作和任务具有情境感知的助手，从而基于这种理解提供定制化辅助。在本文中，我们开发了一个基于多模态大语言模型的上下文感知指令任务助手（InsTALL），它利用在线视觉流（例如用户的屏幕共享或视频录制）并实时响应用户关于当前任务的查询。为了提供有用的辅助，InsTALL 1）在任务视频和配对的文本数据上训练多模态模型，以及 2）从视频数据中自动提取任务图，并在训练和推理时利用该任务图。我们展示了 InsTALL 在提出的多模态活动理解子任务——任务识别（TR）、动作识别（AR）、下一步动作预测（AP）和计划预测（PP）——上达到了最先进的性能，并在两个与自动错误识别相关的新子任务上优于现有基线。
英文摘要：
The improved competence of generative models can help building multi-modal virtual assistants that leverage modalities beyond language. By observing humans performing multi-step tasks, one can build assistants that have situational awareness of actions and tasks being performed, enabling them to cater assistance based on this understanding. In this paper, we develop a Context-aware Instructional Task Assistant with Multi-modal Large Language Models (InsTALL) that leverages an online visual stream (e.g. a user's screen share or video recording) and responds in real-time to user queries related to the task at hand. To enable useful assistance, InsTALL 1) trains a multi-modal model on task videos and paired textual data, and 2) automatically extracts task graph from video data and leverages it at training and inference time. We show InsTALL achieves state-of-the-art performance across proposed sub-tasks considered for multimodal activity understanding -- task recognition (TR), action recognition (AR), next action prediction (AP), and plan prediction (PP) -- and outperforms existing baselines on two novel sub-tasks related to automatic error identification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
