语义感知的动态回顾-前瞻推理用于事件级视频问答
Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering
AI总结：
提出语义感知的动态回顾-前瞻推理方法，利用问题的SRL结构指导跨帧推理，在TrafficQA数据集上超越现有最先进模型。
摘要：
事件级视频问答（EVQA）需要跨视频事件进行复杂推理，以获取提供最佳答案所需的视觉信息。然而，尽管模型性能取得了显著进展，很少有研究关注利用问题与视觉信息之间的显式语义联系，尤其是在事件级别。有必要利用这种语义联系来促进跨视频帧的复杂推理。因此，我们提出了一种用于视频问答的语义感知的动态回顾-前瞻推理方法。具体来说，我们在动态推理过程中显式地使用问题的语义角色标注（SRL）结构，根据当前关注的问题SRL结构中的哪个部分（如施事者、动词、受事者等）来决定是否移动到下一帧。我们在基准EVQA数据集TrafficQA上进行了实验。结果表明，与之前最先进的模型相比，我们提出的方法取得了更优越的性能。我们的代码将公开提供以供研究使用。
英文摘要：
Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers. However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level. There is need for using such semantic connections to facilitate complex reasoning across video frames. Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on. We conduct experiments on a benchmark EVQA dataset - TrafficQA. Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models. Our code will be made publicly available for research use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
