arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.01447cs.MMcs.CLcs.CV
收藏

基于Generator-Pretester网络的端到端视频应答生成

End-to-End Video Question-Answer Generation with Generator-Pretester Network

National Taiwan University(国立台湾大学);University of Toronto(多伦多大学)

Hung-Ting Su, Chen-Hsi Chang, Po-Wei Shen, Yu-Siang Wang, Ya-Liang Chang, Yu-Cheng Chang, Pu-Jen Cheng, Winston H. Hsu

摘要:

针对多媒体环境下的视频问答任务,研究了一种新的视频问答生成任务。由于昂贵的数据注释成本,许多广泛使用的大规模视频QA数据集(如Video-QA,MSVD-QA和MSRVTT-QA)使用字幕问题生成(CapQG)自动注释,CapQG输入字幕而不是视频本身。由于字幕既不完全代表视频,也不总是实际可用,因此通过视频问答生成(VQAG)基于视频生成问答对至关重要。现有的视频到文本(V2 T)方法,尽管采用视频作为输入,但仅生成一个问题。在这项工作中,我们提出了一个新的模型Generator-Pretester Network,它专注于两个组件:(1)联合问答生成器(JQAG),它生成一个问题及其相应的答案,以允许视频问题“问答”训练。(2)Pretester(PT)通过尝试回答问题来验证生成的问题,并使用模型的建议答案和地面真实答案检查预测试答案。我们用仅有的两个可用的大规模人类注释的视频QA数据集来评估我们的系统,并实现了最先进的问题生成性能。此外,仅在视频QA任务中使用我们生成的QA对,我们可以超过一些监督基线。我们将生成的问题应用于视频QA应用程序,并仅使用生成的问题超越了一些监督基线。作为一种预训练策略,当使用带注释的数据进行半监督(20%)或全监督学习时,我们的性能优于CapQG和迁移学习方法。这些实验结果为视频问答训练提供了新的视角。

英文摘要:

We study a novel task, Video Question-Answer Generation (VQAG), for challenging Video Question Answering (Video QA) task in multimedia. Due to expensive data annotation costs, many widely used, large-scale Video QA datasets such as Video-QA, MSVD-QA and MSRVTT-QA are automatically annotated using Caption Question Generation (CapQG) which inputs captions instead of the video itself. As captions neither fully represent a video, nor are they always practically available, it is crucial to generate question-answer pairs based on a video via Video Question-Answer Generation (VQAG). Existing video-to-text (V2T) approaches, despite taking a video as the input, only generate a question alone. In this work, we propose a novel model Generator-Pretester Network that focuses on two components: (1) The Joint Question-Answer Generator (JQAG) which generates a question with its corresponding answer to allow Video Question "Answering" training. (2) The Pretester (PT) verifies a generated question by trying to answer it and checks the pretested answer with both the model's proposed answer and the ground truth answer. We evaluate our system with the only two available large-scale human-annotated Video QA datasets and achieves state-of-the-art question generation performances. Furthermore, using our generated QA pairs only on the Video QA task, we can surpass some supervised baselines. We apply our generated questions to Video QA applications and surpasses some supervised baselines using generated questions only. As a pre-training strategy, we outperform both CapQG and transfer learning approaches when employing semi-supervised (20%) or fully supervised learning with annotated data. These experimental results suggest the novel perspectives for Video QA training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑