VIGiA：通过对话推理和检索进行教学视频指导
VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval
AI总结：
VIGiA是一种多模态对话模型，通过对话推理和检索处理复杂的教学视频行动计划，实现基于视觉输入和计划的对话理解，优于现有方法，在计划感知的VQA任务中达到90%以上准确率。
摘要：
我们引入了VIGiA，一种新的多模态对话模型，旨在理解和推理复杂的多步骤教学视频行动计划。与以往仅关注文本指导或孤立处理视觉和语言的工作不同，VIGiA支持基于计划的对话，需要推理视觉输入、教学计划和交错的用户交互。为此，VIGiA集成了两项关键能力：（1）多模态计划推理，使模型能够将单模态和多模态查询与当前任务计划对齐并准确响应；（2）基于计划的检索，允许其在文本或视觉表示中检索相关计划步骤。实验在包含丰富教学视频对话的新型数据集上进行，该数据集与烹饪和DIY计划对齐。我们的评估表明，VIGiA在对话计划指导设置中优于现有最先进模型，在计划感知的VQA任务中达到90%以上的准确率。
英文摘要：
We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans. Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user interactions. To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni- and multimodal queries with the current task plan and respond accurately; and (2) plan-based retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations. Experiments were done on a novel dataset with rich Instructional Video Dialogues aligned with Cooking and DIY plans. Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90\% accuracy on plan-aware VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
