面向事件驱动活动的基于视频的对话数据集与评估指标
A Video-grounded Dialogue Dataset and Metric for Event-driven Activities
AI总结：
本文提出了VDAct数据集和VDEval评估指标，通过包含更长更复杂的视频和基于会话上下文的评估，解决了事件驱动活动中视频对话理解的挑战。
摘要：
本文提出了VDAct，一个用于事件驱动活动的基于视频对话的数据集，以及VDEval，一个专为该任务设计的基于会话的上下文评估指标。与现有数据集不同，VDAct包含更长、更复杂的视频，描绘了各种事件驱动的活动，这些活动需要高级的上下文理解才能生成准确的响应。该数据集包含3,000个对话和超过30,000个问答对，源自1,000个具有不同活动场景的视频。由于活动场景的广泛谱系和问题类型的广泛范围，VDAct显示出极具挑战性的特征。对最先进的视觉基础模型的实证研究突出了它们在处理我们数据集上某些问题类型时的局限性。此外，VDEval整合了对话会话历史和从我们补充的知识图谱中提取的视频内容摘要来评估单个响应，在VDAct数据集上与人工评估的相关性显著高于仅依赖单个对话上下文的现有评估指标。
英文摘要：
This paper presents VDAct, a dataset for a Video-grounded Dialogue on Event-driven Activities, alongside VDEval, a session-based context evaluation metric specially designed for the task. Unlike existing datasets, VDAct includes longer and more complex video sequences that depict a variety of event-driven activities that require advanced contextual understanding for accurate response generation. The dataset comprises 3,000 dialogues with over 30,000 question-and-answer pairs, derived from 1,000 videos with diverse activity scenarios. VDAct displays a notably challenging characteristic due to its broad spectrum of activity scenarios and wide range of question types. Empirical studies on state-of-the-art vision foundation models highlight their limitations in addressing certain question types on our dataset. Furthermore, VDEval, which integrates dialogue session history and video content summaries extracted from our supplementary Knowledge Graphs to evaluate individual responses, demonstrates a significantly higher correlation with human assessments on the VDAct dataset than existing evaluation metrics that rely solely on the context of single dialogue turns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
