从视频到对话：任务协助的视角指令
From Videos to Conversations: Egocentric Instructions for Task Assistance
AI总结：
本文提出从视频自动转换为多模态任务指导对话的框架，并引入HowToDIV数据集，用于多模态任务协助的基准测试。
摘要：
许多日常任务，从家电维修和烹饪到汽车维护，都需要专业知识，特别是复杂的多步骤过程。尽管对增强现实（AR）辅助中的人工智能代理日益感兴趣，但进展仍然受限于大规模多模态对话数据集的稀缺，部分原因是人类辅助数据收集的成本和物流复杂性。在本文中，我们提出了一种框架，可自动将单人指导视频转换为两人多模态任务指导对话。我们的全自动流程基于大型语言模型，提供了一种可扩展且成本效益高的替代传统数据收集方法。使用此框架，我们引入了HowToDIV多模态数据集，包含507次对话、6,636个问题回答对和24小时的视频，涵盖多个领域。每次会话都包含多轮专家-新手互动。最后，我们使用Gemma 3和Qwen 2.5在HowToDIV上报告了基线结果，为多模态任务协助提供了初始基准。
英文摘要：
Many everyday tasks, ranging from appliance repair and cooking to car maintenance, require expert knowledge, particularly for complex, multi-step procedures. Despite growing interest in AI agents for augmented reality (AR) assistance, progress remains limited by the scarcity of large-scale multimodal conversational datasets grounded in real-world task execution, in part due to the cost and logistical complexity of human-assisted data collection. In this paper, we present a framework to automatically transform single person instructional videos into two-person multimodal task-guidance conversations. Our fully automatic pipeline, based on large language models, provides a scalable and cost efficient alternative to traditional data collection approaches. Using this framework, we introduce HowToDIV, a multimodal dataset comprising 507 conversations, 6,636 question answer pairs, and 24 hours of video spanning multiple domains. Each session consists of a multi-turn expert-novice interaction. Finally, we report baseline results using Gemma 3 and Qwen 2.5 on HowToDIV, providing an initial benchmark for multimodal procedural task assistance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
