arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.09105cs.CVcs.CL
收藏

面向视频问答的位置感知图卷积网络

Location-aware Graph Convolutional Networks for Video Question Answering

South China University of Technology(华南理工大学);Peng Cheng Laboratory, Shenzhen(鹏城实验室);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)

Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, Chuang Gan

摘要:

我们解决了视频问答的挑战性任务,这需要机器以自然语言的形式回答有关视频的问题。现有的方法试图在视频帧特征上应用时空注意力机制,而没有明确地对视频中发生的对象交互之间的位置和关系进行建模。然而,对象交互与其位置信息之间的关系对于动作识别和问题推理都是非常关键的。在这项工作中,我们建议将一个对象的位置信息的图形结构中的视频内容表示为一个位置感知的图形。在这里,每个节点都与由其外观和位置特征表示的对象相关联。基于所构造的图,我们提出使用图卷积来推断动作的类别和时间位置。由于图是建立在对象上的,我们的方法能够专注于前景动作内容,以获得更好的视频问答。最后,我们利用注意力机制将图卷积的输出与编码的问题特征相结合,用于最终答案推理。大量的实验证明了所提出的方法的有效性。具体来说,我们的方法在TGIF-QA、Youtube 2 Text-QA和MSVD-QA数据集上的表现明显优于最先进的方法。代码和预训练模型可在以下网址公开获取:https://github.com/SunDoge/L-GCN

英文摘要:

We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art methods attempt to apply spatio-temporal attention mechanism on video frame features without explicitly modeling the location and relations among object interaction occurred in videos. However, the relations between object interaction and their location information are very critical for both action recognition and question reasoning. In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction. Here, each node is associated with an object represented by its appearance and location features. Based on the constructed graph, we propose to use graph convolution to infer both the category and temporal locations of an action. As the graph is built on objects, our method is able to focus on the foreground action contents for better video question answering. Lastly, we leverage an attention mechanism to combine the output of graph convolution and encoded question features for final answer reasoning. Extensive experiments demonstrate the effectiveness of the proposed methods. Specifically, our method significantly outperforms state-of-the-art methods on TGIF-QA, Youtube2Text-QA, and MSVD-QA datasets. Code and pre-trained models are publicly available at: https://github.com/SunDoge/L-GCN

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑